[PATCH bpf-next v3 00/51] bpf: use scalar evolution to widen bounded loops

[PATCH bpf-next v3 00/51] bpf: use scalar evolution to widen bounded loops

[Eduard Zingerman]

This series implements the scalar evolution (SCEV) technique for
verification of loops.

Scalar evolution is a static analysis technique that infers algebraic
expressions describing how variables change within a loop body.
These expressions can then be used to estimate the number of loop body
executions. This estimate can be used to represent induction variables
as ranges instead of enumerating each possible value.

For example, the following loop:

  for (r0 = 0; r0 < 10; r0++) {
    ...
  }

would now be verified with the assumption that r0 is a scalar value
in the range [0..9] within its body.

See the following patches for a more technical overview:
- "bpf: compute scalar evolution expressions for loops"
- "bpf: use SCEV to widen bounded loops"

The series can be viewed as consisting of the following parts:
- Preparatory patches fixing and extending liveness analysis and
  adding utility functions. The liveness changes collect stack access
  effects and register dependencies, including:
  - "bpf: fix arg_track_xfer() for atomic stack spills and fills"
  - "bpf: derive global subprog stack liveness from BTF signatures"
  - "bpf: compute parameters liveness in bpf_compute_live_registers()"
- Patches relaxing the verifier's current restrictions on accessing
  memory via varying offset pointers, specifically:
  - "bpf: allow subrange relations for PTR_TO_STACK in regsafe()"
  - "bpf: representation for intervals with steps"
  - "bpf: varying offset access support for PTR_TO_BTF_ID pointers"
- Patches to compute the immediate dominator tree and loop hierarchy.
- Patches implementing scalar evolution and integrating it with
  the main verification pass:
  - "bpf: allow precision backtracking between overlapping checkpoints"
  - "bpf: compute scalar evolution expressions for loops"
  - "bpf: use SCEV to widen bounded loops"
  - "bpf: avoid widening registers that hinder exact stack-slot tracking"
- Tests and test harness updates.

Literature
==========

- "Symbolic Evaluation of Chains of Recurrences for Loop Optimization"
  Robert A. van Engelen, 2000
- "The CR# Algebra and its Application in Loop Analysis and Optimization"
  Robert A. van Engelen, 2004

Supported patterns
==================

As scalar evolution analysis attempts to infer recurring relationships
from algebraic expressions describing loop variables, there are some
limitations on what can be expressed.

Below is a list of patterns that are expected to work with the current
implementation. The examples are pseudo-code for the indicated BPF
instruction shapes.

Loop widening currently requires allow_uninit_stack, the verifier mode
that permits reading uninitialized stack bytes. This requirement applies
even to loops that use only registers. Without it, the verifier uses
its existing iteration-by-iteration checking.

1. Counted loops, including decreasing counters:

       u64 i = 0;
       do { i += 1; } while (i < 10);

       u64 j = 3;
       do { j += -1; } while (j != 0);

   Header ranges are i in [0,9] and j in [1,3].
   Pre-condition loops, such as while (i < 10) { i += 1; },
   are also supported. The counter can reside in a register or a fixed
   8-byte frame-pointer spill.

2. Strided accesses into a map value:

       /* bytes is a map-value array with at least 16 bytes. */
       u64 i = 0, off = 0;
       do {
           bytes[off] = 1;
           i += 1;
           off += 2;
       } while (i < 8);

   At the header, off is in [0,14] with step 2. The byte store stays
   within the map value even though i and off are tracked separately.

3. A nonconstant entry value with a separate counted induction variable:

       u64 x = bpf_get_prandom_u32() & 6;  // {0,2,4,6}
       u64 i = 0;
       do { x += 6; i += 1; } while (i < 3);

   The counter i supplies the bound. At the header, x widens to [0,18]
   with step 2, retaining the alignment common to its entry values and
   its slope; x itself need not start at a constant.

4. Conditional assignment from an invariant value:

       u64 x = 5;
       for (u64 i = 0; i < 3; i += 1)
           if (i == 1) x = 10;
       return x;

   Widening joins the pre-loop value 5 with the assigned value 10,
   giving x in [5,10] both inside and after the loop.

5. Early exits in addition to the counted backedge:

       u64 i = 0;
       do {
           if (bpf_get_prandom_u32() == 0) break;
           i += 1;
       } while (i != 3);

   The counter still bounds the loop. An additional exit reduces the
   available lower-bound information; it does not by itself prevent
   widening.

6. Array accesses through different memory types:

   Bounded varying offsets are supported for arrays in BTF-typed
   objects, stack memory, map values and keys, packet data and
   metadata, and memory buffers. The usual bounds, alignment and
   access restrictions still apply.

   For stack arrays indexed by a widened loop counter, only byte and
   half-word accesses are supported; see limitation 5 below.

   For an array in a BTF-typed object:

       struct inner { int a, b; };
       struct object { struct inner arr[8]; };
       /* o points to a live, non-NULL BTF-typed struct object. */
       u64 i = bpf_get_prandom_u32() & 7;
       value = o->arr[i].b;

   The 4-byte load has offsets 4 + 8*i, i in [0,7]. Bounds keep
   accesses within arr; base and step 8 ensure that each possible
   offset selects member b. Such accesses also work without a loop.

7. Nested loops with separate counters:

       u64 i = 0;
       do {
           u64 j = 0;
           do {
               /* body */
               j += 1;
           } while (j < 3);
           i += 1;
       } while (i < 4);

   Both loops can be widened: i is in [0,3] at the outer header,
   and j is in [0,2] at the inner header. The inner counter is reset
   on each outer iteration; i is unchanged by the inner loop.

8. Incrementing and comparing a pointer directly:

       /* bytes is a map-value array with at least 8 bytes. */
       u8 *p = bytes, *end = bytes + 8;
       do {
           *p = 1;
           p++;
       } while (p != end);

   The initial pointer and fixed end pointer give eight iterations,
   without a separate integer counter. At the loop header, p ranges
   from bytes to bytes + 7, so each byte store stays within the array.
   Deriving the iteration count requires the compared pointers to have
   a provable common memory origin.

9. A loop bounded by an exit other than the latch:

       u64 i = 0, len = 14;
       do {
           if (i == len)
               break;
           use(i);
           i += 1;
       } while (i != 64);

   Every exit condition dominating the latch bounds the iteration
   count; the smallest bound wins. Here i is in [0,14] at the header,
   not [0,63], so use(i) sees the range established by the break.

Limitations
===========

Patterns that lose precision after widening
-------------------------------------------

Widening can lose information that the verifier gets by checking
each iteration separately. For example:

    r7 = 5;
    for (r6 = 0; r6 < 3; r6++)
        if (r6 == 1)
            r7 = 10;
    if (r7 != 10)
        invalid_stack_read();

R7 is always 10 at the real exit, but is represented as [5, 10] at
loop entry and after the loop. The verifier therefore cannot exclude
the invalid read and rejects this otherwise safe program.
It would be hard to avoid this limitation.

The same issue affects correlated linear counters at loop exit:

    u64 i = 0, off = 0;
    while (i < 8) {
        i++;
        off += 2;
    }
    if (off != 16)
        invalid_stack_read();

Enumeration establishes off == 16. Widening tracks i and off
independently, and the exit check on i does not reconstruct off's exact
final value. This can be lifted after analysis adjustments.

Patterns not widened yet
------------------------

If one of the variables modified in the loop cannot be widened, the
verifier checks each iteration instead. Examples include:

1. 32-bit recurrences and comparisons:

       for (u32 i = 0; i < 8; i++)      // ALU32 / JMP32
           use(i);

   Only 64-bit arithmetic and comparisons are supported.
   This limitation will be lifted.

2. Non-linear updates:

       for (u64 i = 1; i < 16; i *= 2)  // non-linear recurrence
           use(i);

   Recognition is currently limited to simple additive recurrences.
   Literature describes a way to represent this, but it is not
   considered a priority at the moment.

3. A controlling counter or bound that is not constant at loop entry:

       u64 limit = unknown_in_range(1, 8);
       for (u64 i = 0; i < limit; i++)
           use(i);

       u64 i = unknown_in_range(0, 7);
       while (i < 8)
           i++;

   Both loops have finite bounds, but iteration-count evaluation
   currently requires concrete entry values. Other induction variables
   may have nonconstant entry ranges once a separate counter establishes
   the iteration bound. This limitation can be lifted eventually.

4. Alternative values referring to another evolving register:

       r7 = 0;
       for (r6 = 0; r6 < 4; r6++)
           if (condition)
               r7 = r6;
       use(r7);

   R6 changes on each iteration. Widening currently supports
   conditional assignments only from constants or values that do not
   change in the loop. This limitation can be lifted.

5. Stack addresses whose widening would lose precise slot tracking:

       u64 slots[4];
       for (u64 i = 0; i < 4; i++)
           slots[i] = i;

       struct bpf_dynptr dptrs[4];
       for (u64 i = 0; i < 4; i++)
           bpf_dynptr_from_xdp(ctx, 0, &dptrs[i]);

   Keep such indices concrete for 4- and 8-byte stack loads/stores and
   calls requiring fixed-offset stack objects. Dependencies inside
   nested loops also constrain the outer loop.
   It would be hard to lift this limitation.

6. Values modified by a nested loop do not get a closed-form summary
   for the enclosing loop:

       u64 sum = 0;
       for (u64 i = 0; i < 4; i++)
           for (u64 j = 0; j < 4; j++)
               sum++;
       use(sum);

   The inner loop may widen, but the outer analysis forgets values
   modified by it rather than deriving the combined recurrence.
   This limitation can be lifted.

7. Multiple backedges and irreducible control flow:

       i = 0;
   H:  if (i >= 8) goto done;
       if (condition) { i++; goto H; }  // first backedge
       i++; goto H;                    // second backedge
   done:

       i = 0;
       if (condition) goto body;       // second entry into the loop
   H:  i++;
   body:
       if (i < 8) goto H;

   Widening requires a reducible loop with one backedge and a supported
   dominating exit condition. Multiple exits are supported, but their
   early-exit paths reduce what can be inferred about iteration counts.
   It is unclear whether this can be lifted at the moment.

8. Large slopes and unsupported iteration bounds:

       for (u64 i = 0, x = 0; i < 4; i++) {
           use(x);
           x += 65536;                // slope outside signed 16 bits
       }

   Slopes used for linear widening must be nonzero signed-16-bit
   constants. The inferred number of header executions must be finite
   and less than U32_MAX. Counter wraparound must be ruled out while
   deriving the bound.

9. Incompatible pointer alternatives or comparison operands:

       p = &map_a[0];
       q = &map_b[0];

       for (u64 i = 0; i < 10; i++) {
           if (i == 5)
               p = q;
           use(*p);
       }

   At the loop header, p has SCEV ANY(p_entry, q). The alternatives
   point into different maps, so their offsets cannot be joined while
   retaining one map's metadata. Joining invariant pointer
   alternatives requires compatible pointer metadata, including type,
   origin IDs, frame number.

10. Some hard-coded constant limits:
   - Programs with more than 16 nested loops in one function are rejected.
   - Loops with more than 256 exits exceed the metadata limits and are
     not analyzed for widening.
   - At most 4 exit conditions dominating the latch are consulted for
     the iteration count of a loop.
   - Expression traversal is limited to depth 8. Only a small fixed number
     of alternative values can be joined.

Veristat changes
================

The main impact of the series is a 23% reduction in number of
processed instructions on a subset of Meta BPF programs.

Verdict changes
---------------

Two programs flip succes->failure, both selftests whose expectations
are updated by the series:
- verifier_gotox/gotox_edges_at_limit: test's CFG is a nest of ~1000
  irreducible loops, deeper than the per-function loop stack (16),
  the program is rejected with "Too many nested loops".
- verifier_scev/conditional_assignment_on_second_iteration:
  a value assigned on the second iteration is checked after the loop
  with its widened range, so a stack read guarded by that check
  is rejected.

No program from the production corpora changes verdict.

Regressions
-----------

20 programs regress by 30-33% and 6 more by 10-21%, all from an
appliction objects sharing one unwinder library.
All programs share a common reason for regression:
- a big outer loop contains several smaller nested loops;
- nested loops are now widened and coverge, their convergence
  adds backedges to the outer loop's SCC, thus flipping
  checkpoint comparisons with the outer loop to RANGE_WITHIN mode.

Support for 32-bit recurrences and comparisons widens these loops too
and removes the regression. This would be handled in a followup.

Statistics
----------

A: master; B: scev-minimal-v3. Corpora: cilium, meta, scx, selftests.
Insns counts verifier-processed instructions, not program size.
Object exclusions: verifier_scev.bpf.o.

Histogram: selected programs, without the table filters.

Total progs: 7952
Old success: 5637
New success: 5646
total_insns diff min:  -99.84%
total_insns diff max:   61.90%
0 -> value: 0
value -> 0: 0
total_insns abs max old: 837,487
total_insns abs max new: 837,487
-100 .. -95  %: 15
 -95 .. -90  %: 12
 -90 .. -85  %: 10
 -85 .. -75  %: 8
 -75 .. -70  %: 6
 -70 .. -65  %: 14
 -65 .. -60  %: 38
 -60 .. -55  %: 10
 -55 .. -50  %: 17
 -50 .. -45  %: 36
 -45 .. -40  %: 9
 -40 .. -35  %: 25
 -35 .. -25  %: 10
 -25 .. -15  %: 11
 -15 .. -10  %: 48
 -10 .. -5   %: 47
  -5 .. 0    %: 111
   0 .. 5    %: 7488
   5 .. 15   %: 3
  15 .. 25   %: 5
  30 .. 35   %: 20
  35 .. 40   %: 7
  40 .. 50   %: 1
  60 .. 65   %: 1

Tables: success in both runs; baseline Insns >= 5,000.
Ranked by percentage change, then instruction-count change, file, program.
Meta names are anonymized; numbered objects are distinct files.

Top wins (50)
-------------

File                                Program                            Insns (A)  Insns (B)       Insns (DIFF)
----------------------------------  ---------------------------------  ---------  ---------  -----------------
verifier_bounds.bpf.o               crossing_64_bit_signed_boundary_2      12330         20   -12310 (-99.84%)
verifier_iterating_callbacks.bpf.o  test1                                   7004         12    -6992 (-99.83%)
bpf_iter_tasks.bpf.o                dump_task_sleepable                    85522        556   -84966 (-99.35%)
bpf_iter_task_stack.bpf.o           dump_task_stack                         7209         65    -7144 (-99.10%)
profiler-10.bpf.o                   hhvm_stack                             23828        347   -23481 (-98.54%)
bpf_iter_unix.bpf.o                 dump_unix                               9026        226    -8800 (-97.50%)
meta-app-22.bpf.o                   program-03                              8705        319    -8386 (-96.34%)
meta-app-22.bpf.o                   program-06                              8948        328    -8620 (-96.33%)
security-sandbox-10.bpf.o           fork                                   30760       1249   -29511 (-95.94%)
profiler-02.bpf.o                   on_py_event                            79064       3864   -75200 (-95.11%)
profiler-03.bpf.o                   on_py_event                            79064       3864   -75200 (-95.11%)
profiler-05.bpf.o                   on_py_event                            79064       3864   -75200 (-95.11%)
security-monitor-66.bpf.o           mac_sysctl_write                        7333        448    -6885 (-93.89%)
profiler-04.bpf.o                   get_task_stacks                        73176       6050   -67126 (-91.73%)
security-monitor-65.bpf.o           lsm_bprm_creds_for_exec                17538       1701   -15837 (-90.30%)
security-monitor-65.bpf.o           lsm_task_alloc                          5674        565    -5109 (-90.04%)
security-monitor-65.bpf.o           lsm_fo_task_update                     11399       1181   -10218 (-89.64%)
security-sandbox-29.bpf.o           test_glob_match                        47257       5477   -41780 (-88.41%)
load-balancer-17.bpf.o              edgeloom_tc                            24854       2984   -21870 (-87.99%)
profiler-13.bpf.o                   handle_aten_op_call_start               7183        976    -6207 (-86.41%)
security-monitor-65.bpf.o           lsm_file_open                         154412      21168  -133244 (-86.29%)
security-monitor-17.bpf.o           test_execve                           293300      46071  -247229 (-84.29%)
security-sandbox-13.bpf.o           kvm_die                                36514       5807   -30707 (-84.10%)
security-monitor-97.bpf.o           child_iter                             41266       8182   -33084 (-80.17%)
scheduler-30.bpf.o                  lavd_dispatch                         290915      66359  -224556 (-77.19%)
security-monitor-25.bpf.o           syscalls_kill                          98038      27912   -70126 (-71.53%)
security-sandbox-14.bpf.o           kernel_module_request                   7880       2357    -5523 (-70.09%)
security-sandbox-14.bpf.o           kernel_load_data                        7881       2358    -5523 (-70.08%)
security-sandbox-14.bpf.o           kernel_read_file                        7882       2359    -5523 (-70.07%)
xdp_lb_bench.bpf.o                  xdp_lb_bench                           27126       9208   -17918 (-66.05%)
firewall-05.bpf.o                   ingress                                54034      18584   -35450 (-65.61%)
firewall-05.bpf.o                   tc_in                                  54034      18584   -35450 (-65.61%)
firewall-07.bpf.o                   ingress                                54034      18584   -35450 (-65.61%)
firewall-07.bpf.o                   tc_in                                  54034      18584   -35450 (-65.61%)
security-sandbox-01.bpf.o           bpf                                    30692      10586   -20106 (-65.51%)
security-monitor-34.bpf.o           syscall_setresuid                     122200      43305   -78895 (-64.56%)
security-monitor-34.bpf.o           syscalls_setfsgid                     122200      43305   -78895 (-64.56%)
security-monitor-34.bpf.o           syscalls_setfsuid                     122200      43305   -78895 (-64.56%)
security-monitor-34.bpf.o           syscalls_setgid                       122200      43305   -78895 (-64.56%)
security-monitor-34.bpf.o           syscalls_setregid                     122200      43305   -78895 (-64.56%)
security-monitor-34.bpf.o           syscalls_setreuid                     122200      43305   -78895 (-64.56%)
security-monitor-34.bpf.o           syscalls_setuid                       122200      43305   -78895 (-64.56%)
security-monitor-15.bpf.o           cve_2021_4034                         122481      43629   -78852 (-64.38%)
security-monitor-16.bpf.o           test                                  122517      43665   -78852 (-64.36%)
security-sandbox-10.bpf.o           exec                                   12908       4626    -8282 (-64.16%)
security-monitor-12.bpf.o           connect_security_socket_connect       216252      78013  -138239 (-63.92%)
security-monitor-19.bpf.o           kernel_modules_do_init                 92980      33776   -59204 (-63.67%)
security-monitor-23.bpf.o           enter_pivot_root                       93518      34311   -59207 (-63.31%)
security-monitor-08.bpf.o           bpf_prog_detect                        93384      34494   -58890 (-63.06%)
firewall-05.bpf.o                   egress                                 58436      21951   -36485 (-62.44%)

Top losses (38)
---------------

File                              Program                        Insns (A)  Insns (B)      Insns (DIFF)
--------------------------------  -----------------------------  ---------  ---------  ----------------
profiler-26.bpf.o                 handle_cu_mem_address_reserve      34326      45724  +11398 (+33.21%)
profiler-26.bpf.o                 handle_cu_mem_create               34326      45724  +11398 (+33.21%)
profiler-26.bpf.o                 handle_malloc                      34326      45724  +11398 (+33.21%)
profiler-26.bpf.o                 handle_cu_mem_map                  34524      45922  +11398 (+33.01%)
profiler-19.bpf.o                 sched_waking                       17454      23153   +5699 (+32.65%)
profiler-27.bpf.o                 handle_cuda_launch_exc             17477      23176   +5699 (+32.61%)
profiler-27.bpf.o                 handle_cuda_launch                 17490      23189   +5699 (+32.58%)
profiler-12.bpf.o                 on_custom_event_begin              17697      23396   +5699 (+32.20%)
profiler-12.bpf.o                 on_custom_event_end                17701      23400   +5699 (+32.20%)
profiler-26.bpf.o                 handle_mtia_cached_alloc           35494      46892  +11398 (+32.11%)
profiler-26.bpf.o                 handle_mtia_allocate_enter         35555      46953  +11398 (+32.06%)
profiler-26.bpf.o                 handle_pytorch_malloc              35847      47245  +11398 (+31.80%)
profiler-19.bpf.o                 oncpu                              17990      23689   +5699 (+31.68%)
profiler-12.bpf.o                 on_task_enqueued                   18035      23734   +5699 (+31.60%)
profiler-12.bpf.o                 sched_waking                       18070      23769   +5699 (+31.54%)
profiler-12.bpf.o                 on_task_dequeued                   18210      23909   +5699 (+31.30%)
profiler-12.bpf.o                 on_request_context_event           18301      24000   +5699 (+31.14%)
profiler-12.bpf.o                 on_pytorch_event                   18302      24001   +5699 (+31.14%)
profiler-12.bpf.o                 on_task_stats                      18384      24083   +5699 (+31.00%)
profiler-12.bpf.o                 oncpu                              18488      24187   +5699 (+30.83%)
profiler-19.bpf.o                 sched_switch_offcpu                64726      78210  +13484 (+20.83%)
profiler-12.bpf.o                 sched_switch                       33515      40257   +6742 (+20.12%)
profiler-09.bpf.o                 on_custom_event_start              38281      45973   +7692 (+20.09%)
profiler-09.bpf.o                 trace_func_entry                   38281      45973   +7692 (+20.09%)
profiler-12.bpf.o                 get_crochet_walltime_stacks        43473      50215   +6742 (+15.51%)
profiler-24.bpf.o                 get_task_stacks                    94740     103742    +9002 (+9.50%)
test_cls_redirect_subprogs.bpf.o  cls_redirect                       72052      73682    +1630 (+2.26%)
security-sandbox-20.bpf.o         redirect_socket_connect            72423      73913    +1490 (+2.06%)
scheduler-28.bpf.o                arena_alloc_mask                   27159      27175      +16 (+0.06%)
scheduler-30.bpf.o                arena_alloc_mask                   27159      27175      +16 (+0.06%)
scx_cake.bpf.o                    arena_alloc_mask                   28721      28737      +16 (+0.06%)
scx_arena.bpf.o                   arena_alloc_mask                   28734      28750      +16 (+0.06%)
scx_arena_selftests.bpf.o         arena_alloc_mask                   28734      28750      +16 (+0.06%)
scx_chaos.bpf.o                   arena_alloc_mask                   28734      28750      +16 (+0.06%)
scx_p2dq.bpf.o                    arena_alloc_mask                   28734      28750      +16 (+0.06%)
scx_cake.bpf.o                    cake_init_task                     28758      28774      +16 (+0.06%)
scheduler-01.bpf.o                arena_alloc_mask                   29197      29208      +11 (+0.04%)
scheduler-03.bpf.o                arena_alloc_mask                   29197      29208      +11 (+0.04%)

Changelog
=========

v2 -> v3:
TLDR: another bunch of major correctness fixes, plus several improvements
      to avoid veristat-meta failure identified for v2.
- Widen only when allow_uninit_stack is set.
- Mark scalar registers and stack spills used by the selected bounding
  condition's base, step and bound expressions precise before widening.
- Join alternative values only for registers with compatible pointer
  metadata. Include the loop-entry value when a register is assigned an
  invariant in the body, including unconditional assignments.
- Consider up to four exit conditions dominating the backedge and use
  the tightest available iteration bound. Match the induction variable
  in either operand of the comparison.
- Require full spills for linear stack recurrences. Treat partially
  spilled and unallocated stack slots as unknown scalars and all-zero
  slots as zero. Materialize writable stack values safely, growing the
  stack when needed.
- Keep base/step in sync with the bounds as the last step of
  reg_bounds_sync(). Preserve the register state when sign extension
  leaves all possible values unchanged. Check that step is nonzero and
  base is smaller than step.
- Handle latch comparison polarity correctly when the continuing branch
  enters a nested loop. Compute SCEVs only for exits owned by the loop,
  fixing a verifier bug for irreducible inner loops.
- Exclude BPF_F_INNER_MAP single-element arrays from the separate-lookup
  shortcut in bpf_same_memory_origin().
- Fix stack-pointer tracking through load-acquire/store-release and
  atomic exchange/fetch/cmpxchg operations, so accesses through recovered
  stack pointers are included in liveness analysis.
- Derive global subprogram stack effects from BTF argument types, so
  freplace cannot invalidate a loop termination proof based on the
  original callee body.
- Compute argument liveness from callee reads for static subprograms
  and from BTF-declared arguments for globals. Prepare argument metadata
  before liveness analysis, avoiding unnecessarily live registers that
  prevented widening.
- Treat already-unbounded scalars as already widened, allowing unknown
  SCEVs such as stack outputs of probe-read helpers called through a
  subprogram.
- Widen to an unbounded range when computing an induction variable's
  range overflows signed 64-bit arithmetic, retaining the power-of-two
  alignment common to the entry value and slope.
- Fix dominator computation for subprograms whose entry is a loop header:
  initialize the entry as its own dominator during the fixed-point
  computation and skip updates to it. Use overflow-checked predecessor
  allocations.
- Replace the private worklist heap with the kernel min-heap and limit
  expression rewriting to 32 passes before falling back to unknown.
- Add regression coverage for the liveness, stack, pointer, iteration
  bound and widening changes. Fix __msg_next validation when an expected
  message is missing.

v1 -> v2:
TLDR: a bunch of major correctness fixes.
- Introduce arg_access_info to distinguish read, may-write and
  must-write effects of helper/kfunc stack accesses. Handle map-specific
  argument semantics and writes through unknown callbacks.
- Fix base/step tracking across signed overflow, truncation,
  sign/zero extension and linked-register updates.
- Rework iteration-count calculation for signed/unsigned increasing
  and decreasing counters, including counter overflow checks.
  Evaluate the full latch base expression and consistently count
  header executions.
- Substitute all registers in instantiate_header_scevs().
- Track pointer origins when evaluating latch expressions.
  Require compatible operands before deriving iteration bounds.
- Preserve the correct congruence base when widening nonconstant
  entry values, assign fresh IDs to widened packet pointers,
  and exclude nullable pointers from widening.
- Account for values modified on nested-loop exit paths when computing
  invariants. Do not use an inner loop's latch to bound an outer loop.
- Restrict pruning against in-progress SCEV states to terminating
  innermost loops sharing the same entry checkpoint, preventing
  inner-loop pruning from hiding a nonterminating outer loop.
- Rework loop entry/exit handling on the main verification pass.
- Extend fixed-stack-offset widening suppression to ANY expressions
  and all dynptr arguments described by helper/kfunc prototypes.
- Fix BTF array handling: retain bounds checks for fixed arrays inside
  flexible-array elements and preserve array type IDs when element
  types contain typedefs.
- Fix SCEV handling of endian conversions and narrow fills on
  big-endian systems.
- Grow expression storage geometrically and scale the expression hash
  table with program size. Add cancellation/rescheduling checks to
  dominator computation and fix the postorder-number allocation leak.
- Pack bpf_reg_state/bpf_func_state for latter to fit back into
  the 1 KiB allocation bucket, measured SCEV memory consumption
  overhead is 3% now.
- Major additions in the selftests.

v1: https://lore.kernel.org/bpf/20260926-scev-minimal-rebase-v1-0-c8e5ab5ba79f@gmail.com/
v2: https://lore.kernel.org/bpf/20261004-scev-minimal-rebase-v2-0-f6254e29fdf2@gmail.com/
---
Eduard Zingerman (51):
      bpf: represent stack access effects with arg_access_info
      bpf: describe helper stack accesses with arg_access_info
      bpf: describe kfunc stack accesses with arg_access_info
      bpf: track may_write flags in liveness
      bpf: summarize may write stack slots in insn_aux_data
      bpf: summarize live stack slots in insn_aux_data
      bpf: summarize regs that may hold a frame pointer in insn_aux_data
      bpf: record write effects for atomic operations in liveness.c
      bpf: fix arg_track_xfer() for atomic stack spills and fills
      bpf: derive global subprog stack liveness from BTF signatures
      bpf: compute parameters liveness in bpf_compute_live_registers()
      bpf: add tnum_alignment()
      bpf: add cnum{32,64}_union()
      bpf: add cnum64_intersect_linear()
      bpf: expose bpf_rev_opcode() and bpf_flip_opcode()
      bpf: expose bpf_grow_stack_state()
      bpf: expose bpf_verbose_snum()
      bpf: allow subrange relations for PTR_TO_STACK in regsafe()
      bpf: representation for intervals with steps
      bpf: varying offset access support for PTR_TO_BTF_ID pointers
      bpf: save DFS postorder numbers for program instructions
      bpf: move the live-register and SCC printout to a standalone function
      bpf: compute immediate dominators
      bpf: compute loop hierarchy
      bpf: add bpf_set_reg_range()
      bpf: add bpf_mark_reg_known_scalar()
      bpf: add bpf_reg_union() and bpf_reg_union_compatible()
      bpf: add bpf_split_cur_state()
      bpf: add bpf_same_memory_origin()
      bpf: add bpf_spill_size()
      bpf: record basic-block ends in insn_aux_data
      bpf: allow precision backtracking between overlapping checkpoints
      bpf: compute scalar evolution expressions for loops
      bpf: use SCEV to widen bounded loops
      bpf: avoid widening registers that hinder exact stack-slot tracking
      bpf: bpf_func_state size optimization
      selftests/bpf: __msg_next tag for matching messages on consecutive lines
      selftests/bpf: bound UNIX socket path loops by sun_path size
      selftests/bpf: test for stack-pointer subrange pruning
      selftests/bpf: tests for may_write stack-liveness tracking
      selftests/bpf: tests for may_def marks of atomic RMW operations
      selftests/bpf: tests for stack pointer spills/fills through atomics
      selftests/bpf: tests for map special cases in bpf_helper_stack_access_bytes()
      selftests/bpf: tests for BTF-based global subprog stack liveness
      selftests/bpf: test for register liveness of subprogram arguments
      selftests/bpf: tests for register base/step arithmetic
      selftests/bpf: tests for register base/step state pruning
      selftests/bpf: tests for varying offset access to PTR_TO_BTF_ID
      selftests/bpf: tests for loop hierarchy computation
      selftests/bpf: tests for immediate dominator computation
      selftests/bpf: tests for SCEV analysis and loop widening

 include/linux/bpf_verifier.h                       |  192 +-
 include/linux/cnum.h                               |    3 +
 include/linux/tnum.h                               |    2 +
 kernel/bpf/Makefile                                |    2 +-
 kernel/bpf/backtrack.c                             |    5 +
 kernel/bpf/btf.c                                   |  159 +-
 kernel/bpf/cfg.c                                   |   56 +-
 kernel/bpf/cnum.c                                  |   36 +
 kernel/bpf/cnum_defs.h                             |   44 +
 kernel/bpf/const_fold.c                            |    2 +
 kernel/bpf/fixups.c                                |   18 +
 kernel/bpf/liveness.c                              |  522 ++--
 kernel/bpf/log.c                                   |   19 +-
 kernel/bpf/loops.c                                 |  616 ++++
 kernel/bpf/ringbuf.c                               |    4 +-
 kernel/bpf/scev.c                                  | 2761 +++++++++++++++++
 kernel/bpf/states.c                                |  216 +-
 kernel/bpf/tnum.c                                  |   11 +
 kernel/bpf/verifier.c                              |  918 +++++-
 .../testing/selftests/bpf/prog_tests/exceptions.c  |    2 +-
 tools/testing/selftests/bpf/prog_tests/verifier.c  |   10 +
 tools/testing/selftests/bpf/progs/bpf_iter_unix.c  |    2 +-
 tools/testing/selftests/bpf/progs/bpf_misc.h       |    6 +
 .../selftests/bpf/progs/compute_live_registers.c   |   41 +-
 .../selftests/bpf/progs/test_skc_to_unix_sock.c    |    2 +-
 .../selftests/bpf/progs/verifier_bounds_step.c     |  544 ++++
 .../bpf/progs/verifier_btf_array_access.c          |  576 ++++
 tools/testing/selftests/bpf/progs/verifier_gotox.c |    6 +-
 tools/testing/selftests/bpf/progs/verifier_idoms.c |  377 +++
 .../selftests/bpf/progs/verifier_kfunc_uninit.c    |    6 +
 .../selftests/bpf/progs/verifier_live_stack.c      |  485 ++-
 .../selftests/bpf/progs/verifier_loop_hierarchy.c  |  346 +++
 tools/testing/selftests/bpf/progs/verifier_scev.c  | 3180 ++++++++++++++++++++
 .../selftests/bpf/progs/verifier_stack_ptr.c       |   36 +
 tools/testing/selftests/bpf/test_loader.c          |   11 +
 35 files changed, 10800 insertions(+), 416 deletions(-)
---
base-commit: e1d84a37cba984388988d2f1ddc84561413f0db2
change-id: 20260925-scev-minimal-rebase-911799c95dff

[PATCH bpf-next v3 01/51] bpf: represent stack access effects with arg_access_info

[Eduard Zingerman]

The following patches in the series add support for may_write property
inference. To properly track this property there is a need to
distinguish three ways the helpers/kfuncs affect stack:
- read;
- write and destroy all previous stack tracking information for
  affected slots;
- both read and write the stack slots, potentially preserving some of
  the tracking information.

The later property can't be encoded by signed access_byte convention.
Hence, introduce arg_access_info with a size and separate may_read,
may_write and must_write bits. Convert the liveness access recording
paths to use it. Proper may_write calculation and support will be
added in the following patches.

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 include/linux/bpf_verifier.h |  8 ++++
 kernel/bpf/liveness.c        | 96 ++++++++++++++++++++++++++++----------------
 2 files changed, 69 insertions(+), 35 deletions(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index c51083c761cf..d4f32acba236 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1723,6 +1723,14 @@ bool bpf_is_may_goto_insn(struct bpf_insn *insn);
 void bpf_verbose_insn(struct bpf_verifier_env *env, struct bpf_insn *insn);
 bool bpf_get_call_summary(struct bpf_verifier_env *env, struct bpf_insn *call,
 			  struct bpf_call_summary *cs);
+/* Stack effects used for state pruning; must_write implies may_write. */
+struct arg_access_info {
+	u32 size;		/* Maximum extent; U32_MAX if unknown. */
+	u8 may_read:1;		/* Incoming contents or initialization may be needed. */
+	u8 may_write:1;
+	u8 must_write:1;	/* Prior verifier state is destroyed throughout size. */
+};
+
 s64 bpf_helper_stack_access_bytes(struct bpf_verifier_env *env,
 				  struct bpf_insn *insn, int arg,
 				  int insn_idx);
diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index cd9523f69298..7b317fb5ca82 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -1437,17 +1437,14 @@ static void arg_track_xfer(struct bpf_verifier_env *env, struct bpf_insn *insn,
 }
 
 /*
- * Record access_bytes from helper/kfunc or load/store insn.
- *   access_bytes > 0:      stack read
- *   access_bytes < 0:      stack write
- *   access_bytes == S64_MIN: unknown   — conservative, mark [0..slot] as read
- *   access_bytes == 0:      no access
- *
+ * Record reads for every touched half-slot. A definite
+ * write requires full coverage, a known size, and a single possible offset.
  */
 static int record_stack_access_off(struct func_instance *instance, s64 fp_off,
-				   s64 access_bytes, u32 frame, u32 insn_idx)
+				   struct arg_access_info info, u32 frame, u32 insn_idx)
 {
 	s32 slot_hi, slot_lo;
+	int err;
 
 	if (fp_off >= 0)
 		/*
@@ -1456,49 +1453,53 @@ static int record_stack_access_off(struct func_instance *instance, s64 fp_off,
 		 * by the main verifier pass later.
 		 */
 		return 0;
-	if (access_bytes == S64_MIN) {
+
+	if (info.may_read && info.size == U32_MAX) {
 		/* helper/kfunc read unknown amount of bytes from fp_off until fp+0 */
 		slot_hi = (-fp_off - 1) / STACK_SLOT_SZ;
 		return mark_stack_read(instance, frame, insn_idx, 0, slot_hi);
 	}
-	if (access_bytes > 0) {
+	if (info.may_read) {
 		/* Mark any touched slot as use */
 		slot_hi = (-fp_off - 1) / STACK_SLOT_SZ;
-		slot_lo = max_t(s32, (-fp_off - access_bytes) / STACK_SLOT_SZ, 0);
-		return mark_stack_read(instance, frame, insn_idx, slot_lo, slot_hi);
-	} else if (access_bytes < 0) {
+		slot_lo = max_t(s32, (-fp_off - info.size) / STACK_SLOT_SZ, 0);
+		err = mark_stack_read(instance, frame, insn_idx, slot_lo, slot_hi);
+		if (err)
+			return err;
+	}
+	if (info.must_write && info.size != U32_MAX) {
 		/* Mark only fully covered slots as def */
-		access_bytes = -access_bytes;
 		slot_hi = (-fp_off) / STACK_SLOT_SZ - 1;
-		slot_lo = max_t(s32, (-fp_off - access_bytes + STACK_SLOT_SZ - 1) / STACK_SLOT_SZ, 0);
+		slot_lo = max_t(s32, (-fp_off - info.size + STACK_SLOT_SZ - 1) / STACK_SLOT_SZ, 0);
 		return mark_stack_write(instance, frame, insn_idx, slot_lo, slot_hi);
 	}
 	return 0;
 }
 
-/*
- * 'arg' is FP-derived argument to helper/kfunc or load/store that
- * reads (positive) or writes (negative) 'access_bytes' into 'use' or 'def'.
- */
-static int record_stack_access(struct bpf_verifier_env *env, struct func_instance *instance,
+/* Record access through a pointer with a known frame and possibly known offsets. */
+static int record_stack_access(struct bpf_verifier_env *env,
+			       struct func_instance *instance,
 			       const struct arg_track *arg,
-			       s64 access_bytes, u32 frame, u32 insn_idx)
+			       struct arg_access_info info, u32 frame, u32 insn_idx)
 {
 	int i, err;
 
-	if (access_bytes == 0)
+	if (!info.size)
 		return 0;
 	if (arg->off_cnt == 0) {
-		if (access_bytes > 0 || access_bytes == S64_MIN)
-			return mark_stack_read_all(env, instance, frame, insn_idx);
+		if (info.may_read) {
+			err = mark_stack_read_all(env, instance, frame, insn_idx);
+			if (err)
+				return err;
+		}
 		return 0;
 	}
-	if (access_bytes != S64_MIN && access_bytes < 0 && arg->off_cnt != 1)
+	if (info.size != U32_MAX && info.must_write && arg->off_cnt != 1)
 		/* multi-offset write cannot set stack_def */
 		return 0;
 
 	for (i = 0; i < arg->off_cnt; i++) {
-		err = record_stack_access_off(instance, arg->off[i], access_bytes, frame, insn_idx);
+		err = record_stack_access_off(instance, arg->off[i], info, frame, insn_idx);
 		if (err)
 			return err;
 	}
@@ -1533,9 +1534,10 @@ static int record_load_store_access(struct bpf_verifier_env *env,
 				    struct arg_track *at, int insn_idx)
 {
 	struct bpf_insn *insn = &env->prog->insnsi[insn_idx];
+	struct arg_access_info info = {};
 	int depth = instance->depth;
-	s32 sz = bpf_size_to_bytes(BPF_SIZE(insn->code));
 	u8 class = BPF_CLASS(insn->code);
+	bool read = false, write = false;
 	struct arg_track resolved, *ptr;
 	int oi;
 
@@ -1552,23 +1554,26 @@ static int record_load_store_access(struct bpf_verifier_env *env,
 	switch (class) {
 	case BPF_LDX:
 		ptr = &at[insn->src_reg];
+		read = true;
 		break;
 	case BPF_STX:
 		if (BPF_MODE(insn->code) == BPF_ATOMIC) {
 			if (insn->imm == BPF_STORE_REL)
-				sz = -sz;
+				write = true;
+			else
+				read = true;
 			if (insn->imm == BPF_LOAD_ACQ)
 				ptr = &at[insn->src_reg];
 			else
 				ptr = &at[insn->dst_reg];
 		} else {
 			ptr = &at[insn->dst_reg];
-			sz = -sz;
+			write = true;
 		}
 		break;
 	case BPF_ST:
 		ptr = &at[insn->dst_reg];
-		sz = -sz;
+		write = true;
 		break;
 	default:
 		return 0;
@@ -1587,32 +1592,53 @@ static int record_load_store_access(struct bpf_verifier_env *env,
 		ptr = &resolved;
 	}
 
+	info.may_read = read;
+	info.may_write = write;
+	info.must_write = write;
+	info.size = bpf_size_to_bytes(BPF_SIZE(insn->code));
 	if (ptr->frame >= 0 && ptr->frame <= depth)
-		return record_stack_access(env, instance, ptr, sz, ptr->frame, insn_idx);
+		return record_stack_access(env, instance, ptr, info, ptr->frame, insn_idx);
 	if (ptr->frame == ARG_IMPRECISE)
 		return record_imprecise(env, instance, ptr->mask, insn_idx);
 	/* ARG_NONE: not derived from any frame pointer, skip */
 	return 0;
 }
 
+/* Adapt the signed helper/kfunc access size until both producers are converted. */
+static struct arg_access_info stack_access_info(s64 bytes)
+{
+	bool write = bytes < 0 && bytes != S64_MIN && -bytes < U32_MAX;
+
+	return (struct arg_access_info) {
+		.size = bytes == S64_MIN
+			? U32_MAX
+			: min_t(u64, bytes < 0 ? -bytes : bytes, U32_MAX),
+		.may_read = bytes > 0 || bytes == S64_MIN,
+		.may_write = write,
+		.must_write = write,
+	};
+}
+
 static int record_arg_access(struct bpf_verifier_env *env,
 			     struct func_instance *instance,
 			     struct bpf_insn *insn,
 			     struct arg_track *at, int arg_idx,
 			     int insn_idx)
 {
+	struct arg_access_info info;
 	int depth = instance->depth;
 	int frame = at->frame;
 	int err = 0;
-	s64 bytes;
 
 	if (!arg_is_fp(at))
 		return 0;
 
 	if (bpf_helper_call(insn)) {
-		bytes = bpf_helper_stack_access_bytes(env, insn, arg_idx, insn_idx);
+		info = stack_access_info(bpf_helper_stack_access_bytes(env, insn,
+								       arg_idx, insn_idx));
 	} else if (bpf_pseudo_kfunc_call(insn)) {
-		bytes = bpf_kfunc_stack_access_bytes(env, insn, arg_idx, insn_idx);
+		info = stack_access_info(bpf_kfunc_stack_access_bytes(env, insn,
+								      arg_idx, insn_idx));
 	} else {
 		for (int f = 0; f <= depth; f++) {
 			err = mark_stack_read_all(env, instance, f, insn_idx);
@@ -1621,11 +1647,11 @@ static int record_arg_access(struct bpf_verifier_env *env,
 		}
 		return 0;
 	}
-	if (bytes == 0)
+	if (!info.size)
 		return 0;
 
 	if (frame >= 0 && frame <= depth)
-		err = record_stack_access(env, instance, at, bytes, frame, insn_idx);
+		err = record_stack_access(env, instance, at, info, frame, insn_idx);
 	else if (frame == ARG_IMPRECISE)
 		err = record_imprecise(env, instance, at->mask, insn_idx);
 	return err;

-- 
2.53.0

[PATCH bpf-next v3 02/51] bpf: describe helper stack accesses with arg_access_info

[Eduard Zingerman]

Adjust bpf_helper_stack_access_bytes() to return results as a struct
arg_access_info. The result is based on bpf_func_proto->argX_type
value as in the following table:

| Argument flags       | allow_uninit_stack | Meaning              |
|----------------------+--------------------+----------------------|
| MEM_WRITE            | true               | may_read, must_write |
| MEM_WRITE+MEM_UNINIT | true               | must_write           |
| MEM_WRITE            | false              | may_read, must_write |
| MEM_WRITE+MEM_UNINIT | false              | may_read, may_write  |
| 0                    | true/false         | may_read             |

MEM_WRITE now produces must_write when allow_uninit_stack is true:
the verifier discards previous stack knowledge over the affected
range. Previously, liveness only treated MEM_UNINIT this way.

Also use resolve_map_arg_type() to account for bpf_map_peek_elem()
treating its buffer as an input for bloom filters.

Annotate ringbuf dynptr submit and discard arguments with MEM_WRITE.
Their existing OBJ_RELEASE handling already invalidates the dynptr, but
the prototypes need to describe the write as well as the release.

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 include/linux/bpf_verifier.h |  6 ++--
 kernel/bpf/liveness.c        |  5 ++-
 kernel/bpf/ringbuf.c         |  4 +--
 kernel/bpf/verifier.c        | 81 +++++++++++++++++++++++---------------------
 4 files changed, 49 insertions(+), 47 deletions(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index d4f32acba236..5c99c024c309 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1731,9 +1731,9 @@ struct arg_access_info {
 	u8 must_write:1;	/* Prior verifier state is destroyed throughout size. */
 };
 
-s64 bpf_helper_stack_access_bytes(struct bpf_verifier_env *env,
-				  struct bpf_insn *insn, int arg,
-				  int insn_idx);
+struct arg_access_info
+bpf_helper_stack_access_bytes(struct bpf_verifier_env *env,
+			      struct bpf_insn *insn, int arg, int insn_idx);
 s64 bpf_kfunc_stack_access_bytes(struct bpf_verifier_env *env,
 				 struct bpf_insn *insn, int arg,
 				 int insn_idx);
diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index 7b317fb5ca82..353b52031577 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -1604,7 +1604,7 @@ static int record_load_store_access(struct bpf_verifier_env *env,
 	return 0;
 }
 
-/* Adapt the signed helper/kfunc access size until both producers are converted. */
+/* Adapt the signed kfunc access size until its producer is converted. */
 static struct arg_access_info stack_access_info(s64 bytes)
 {
 	bool write = bytes < 0 && bytes != S64_MIN && -bytes < U32_MAX;
@@ -1634,8 +1634,7 @@ static int record_arg_access(struct bpf_verifier_env *env,
 		return 0;
 
 	if (bpf_helper_call(insn)) {
-		info = stack_access_info(bpf_helper_stack_access_bytes(env, insn,
-								       arg_idx, insn_idx));
+		info = bpf_helper_stack_access_bytes(env, insn, arg_idx, insn_idx);
 	} else if (bpf_pseudo_kfunc_call(insn)) {
 		info = stack_access_info(bpf_kfunc_stack_access_bytes(env, insn,
 								      arg_idx, insn_idx));
diff --git a/kernel/bpf/ringbuf.c b/kernel/bpf/ringbuf.c
index 3f1013d80544..447863ece26a 100644
--- a/kernel/bpf/ringbuf.c
+++ b/kernel/bpf/ringbuf.c
@@ -722,7 +722,7 @@ BPF_CALL_2(bpf_ringbuf_submit_dynptr, struct bpf_dynptr_kern *, ptr, u64, flags)
 const struct bpf_func_proto bpf_ringbuf_submit_dynptr_proto = {
 	.func		= bpf_ringbuf_submit_dynptr,
 	.ret_type	= RET_VOID,
-	.arg1_type	= ARG_PTR_TO_DYNPTR | DYNPTR_TYPE_RINGBUF | OBJ_RELEASE,
+	.arg1_type	= ARG_PTR_TO_DYNPTR | DYNPTR_TYPE_RINGBUF | OBJ_RELEASE | MEM_WRITE,
 	.arg2_type	= ARG_ANYTHING,
 };
 
@@ -741,7 +741,7 @@ BPF_CALL_2(bpf_ringbuf_discard_dynptr, struct bpf_dynptr_kern *, ptr, u64, flags
 const struct bpf_func_proto bpf_ringbuf_discard_dynptr_proto = {
 	.func		= bpf_ringbuf_discard_dynptr,
 	.ret_type	= RET_VOID,
-	.arg1_type	= ARG_PTR_TO_DYNPTR | DYNPTR_TYPE_RINGBUF | OBJ_RELEASE,
+	.arg1_type	= ARG_PTR_TO_DYNPTR | DYNPTR_TYPE_RINGBUF | OBJ_RELEASE | MEM_WRITE,
 	.arg2_type	= ARG_ANYTHING,
 };
 
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 353bde9ae227..bbb81ba0f2dc 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -14358,35 +14358,32 @@ int bpf_fetch_kfunc_arg_meta(struct bpf_verifier_env *env,
 }
 
 /*
- * Determine how many bytes a helper accesses through a stack pointer at
- * argument position @arg (0-based, corresponding to R1-R5).
- *
- * Returns:
- *   > 0   known read access size in bytes
- *     0   doesn't read anything directly
- * S64_MIN unknown
- *   < 0   known write access of (-return) bytes
+ * Describe a helper's stack access through argument @arg (0-based, R1-R5).
+ * must_write means the verifier destroys prior state throughout size bytes.
+ * In other words, must_write follows the verifier's model of the
+ * function's behaviour, which is safe to use for is_state_visited() pruning.
  */
-s64 bpf_helper_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn *insn,
-				  int arg, int insn_idx)
+struct arg_access_info
+bpf_helper_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn *insn,
+			      int arg, int insn_idx)
 {
 	struct bpf_insn_aux_data *aux = &env->insn_aux_data[insn_idx];
+	struct arg_access_info info = {
+		.size = U32_MAX,
+		.may_read = true,
+		.may_write = true,
+	};
 	const struct bpf_func_proto *fn;
+	enum bpf_access_type access_type;
 	enum bpf_arg_type at;
-	bool full_write;
-	s64 size;
+	bool exact_size = true;
+	u64 size = U32_MAX;
 
 	if (bpf_get_helper_proto(env, insn->imm, &fn) < 0)
-		return S64_MIN;
+		return info;
 
 	at = fn->arg_type[arg];
-	/*
-	 * Generic outputs may leave bytes untouched. Keep prior initialization
-	 * live when the caller cannot read uninitialized bytes. Constructors of
-	 * special objects, such as dynptrs, still define their storage.
-	 */
-	full_write = (at & MEM_UNINIT) &&
-		     (!arg_type_is_raw_mem(at) || env->allow_uninit_stack);
+	access_type = func_arg_access_type(at);
 
 	switch (base_type(at)) {
 	case ARG_PTR_TO_MAP_KEY:
@@ -14409,6 +14406,10 @@ s64 bpf_helper_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn
 
 		i = aux->const_reg_vals[map_reg];
 		if (i < env->used_map_cnt) {
+			/* Bloom-filter peek reads the value buffer. */
+			if (!is_key && insn->imm == BPF_FUNC_map_peek_elem &&
+			    env->used_maps[i]->map_type == BPF_MAP_TYPE_BLOOM_FILTER)
+				access_type = BPF_READ;
 			size = is_key ? env->used_maps[i]->key_size
 				      : env->used_maps[i]->value_size;
 			goto out;
@@ -14418,7 +14419,12 @@ s64 bpf_helper_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn
 		 * Map pointer is not known at this call site (e.g. different
 		 * maps on merged paths).  Conservatively return the largest
 		 * key_size or value_size across all maps used by the program.
+		 * This is only an upper bound, so it cannot establish a definite
+		 * write. Map-dependent argument types can also turn an output
+		 * into an input, so conservatively retain a read dependency.
 		 */
+		exact_size = false;
+		access_type |= BPF_READ;
 		val = 0;
 		for (i = 0; i < env->used_map_cnt; i++) {
 			struct bpf_map *map = env->used_maps[i];
@@ -14434,7 +14440,7 @@ s64 bpf_helper_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn
 			}
 		}
 		if (!val)
-			return S64_MIN;
+			return info;
 		size = val;
 		goto out;
 	}
@@ -14448,18 +14454,12 @@ s64 bpf_helper_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn
 			int size_reg = BPF_REG_1 + arg + 1;
 
 			if (aux->const_reg_mask & BIT(size_reg)) {
-				size = (s64)aux->const_reg_vals[size_reg];
+				size = aux->const_reg_vals[size_reg];
 				goto out;
 			}
-			/*
-			 * Size arg is const on each path but differs across merged
-			 * paths. Reads may extend anywhere up to the frame top.
-			 */
-			if (full_write)
-				return 0;
-			return S64_MIN;
 		}
-		return S64_MIN;
+		/* Preserve access directions even when the extent is unknown. */
+		goto out;
 	case ARG_PTR_TO_DYNPTR:
 		size = BPF_DYNPTR_SIZE;
 		break;
@@ -14469,18 +14469,21 @@ s64 bpf_helper_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn
 		 * doesn't access stack. The callback subprog does and it's
 		 * analyzed separately.
 		 */
-		return 0;
+		return (struct arg_access_info) {};
 	default:
-		return S64_MIN;
+		return info;
 	}
 out:
-	/*
-	 * Other accesses keep the previous state live, including untouched bytes
-	 * of an unprivileged generic output.
-	 */
-	if (full_write)
-		return -size;
-	return size;
+	info.size = min_t(u64, size, U32_MAX);
+	info.may_read = !!(access_type & BPF_READ);
+	info.may_write = !!(access_type & BPF_WRITE);
+	info.must_write = info.may_write && exact_size && info.size != U32_MAX;
+	/* Generic unprivileged outputs retain prior initialization state. */
+	if (!env->allow_uninit_stack && arg_type_is_raw_mem(at)) {
+		info.may_read = true;
+		info.must_write = false;
+	}
+	return info;
 }
 
 /*

-- 
2.53.0

[PATCH bpf-next v3 03/51] bpf: describe kfunc stack accesses with arg_access_info

[Eduard Zingerman]

Return kfunc stack effects through arg_access_info while preserving the
existing BTF size lookup and read/def classification. Retain __sz/__szk
handling and conservatively mark pointer arguments as possible writes.

Remove the temporary signed-size adapter from liveness. Add liveness
assertions for privileged and unprivileged __uninit outputs.

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 include/linux/bpf_verifier.h                       |  6 +--
 kernel/bpf/liveness.c                              | 18 +--------
 kernel/bpf/verifier.c                              | 43 ++++++++++++----------
 .../selftests/bpf/progs/verifier_kfunc_uninit.c    |  6 +++
 4 files changed, 33 insertions(+), 40 deletions(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 5c99c024c309..eafe79929dfa 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1734,9 +1734,9 @@ struct arg_access_info {
 struct arg_access_info
 bpf_helper_stack_access_bytes(struct bpf_verifier_env *env,
 			      struct bpf_insn *insn, int arg, int insn_idx);
-s64 bpf_kfunc_stack_access_bytes(struct bpf_verifier_env *env,
-				 struct bpf_insn *insn, int arg,
-				 int insn_idx);
+struct arg_access_info
+bpf_kfunc_stack_access_bytes(struct bpf_verifier_env *env,
+			     struct bpf_insn *insn, int arg, int insn_idx);
 int bpf_compute_subprog_arg_access(struct bpf_verifier_env *env);
 
 int bpf_stack_liveness_init(struct bpf_verifier_env *env);
diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index 353b52031577..d4089a488784 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -1604,21 +1604,6 @@ static int record_load_store_access(struct bpf_verifier_env *env,
 	return 0;
 }
 
-/* Adapt the signed kfunc access size until its producer is converted. */
-static struct arg_access_info stack_access_info(s64 bytes)
-{
-	bool write = bytes < 0 && bytes != S64_MIN && -bytes < U32_MAX;
-
-	return (struct arg_access_info) {
-		.size = bytes == S64_MIN
-			? U32_MAX
-			: min_t(u64, bytes < 0 ? -bytes : bytes, U32_MAX),
-		.may_read = bytes > 0 || bytes == S64_MIN,
-		.may_write = write,
-		.must_write = write,
-	};
-}
-
 static int record_arg_access(struct bpf_verifier_env *env,
 			     struct func_instance *instance,
 			     struct bpf_insn *insn,
@@ -1636,8 +1621,7 @@ static int record_arg_access(struct bpf_verifier_env *env,
 	if (bpf_helper_call(insn)) {
 		info = bpf_helper_stack_access_bytes(env, insn, arg_idx, insn_idx);
 	} else if (bpf_pseudo_kfunc_call(insn)) {
-		info = stack_access_info(bpf_kfunc_stack_access_bytes(env, insn,
-								      arg_idx, insn_idx));
+		info = bpf_kfunc_stack_access_bytes(env, insn, arg_idx, insn_idx);
 	} else {
 		for (int f = 0; f <= depth; f++) {
 			err = mark_stack_read_all(env, instance, f, insn_idx);
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index bbb81ba0f2dc..8987c9a69d10 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -14487,28 +14487,29 @@ bpf_helper_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn *ins
 }
 
 /*
- * Determine how many bytes a kfunc accesses through a stack pointer at
- * argument position @arg (0-based, corresponding to R1-R5).
- *
- * Returns:
- *   > 0      known read access size in bytes
- *     0      doesn't access memory through that argument (ex: not a pointer)
- *   S64_MIN  unknown
- *   < 0      known write access of (-return) bytes
+ * Describe a kfunc's stack access through argument slot @arg (0-based).
+ * As for helpers, must_write describes destruction of prior verifier state,
+ * rather than a guarantee that the callee writes every byte at runtime.
  */
-s64 bpf_kfunc_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn *insn,
-				 int arg, int insn_idx)
+struct arg_access_info
+bpf_kfunc_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn *insn,
+			     int arg, int insn_idx)
 {
 	struct bpf_insn_aux_data *aux = &env->insn_aux_data[insn_idx];
+	struct arg_access_info info = {
+		.size = U32_MAX,
+		.may_read = true,
+		.may_write = true,
+	};
 	struct bpf_call_arg_meta meta;
 	const struct btf_param *args;
 	const struct btf_type *t, *ref_t;
 	const struct btf *btf;
 	u32 i, slot, nargs, type_size;
-	s64 size;
+	u64 size;
 
 	if (bpf_fetch_kfunc_arg_meta(env, insn->imm, insn->off, &meta) < 0)
-		return S64_MIN;
+		return info;
 
 	btf = meta.btf;
 	args = btf_params(meta.func_proto);
@@ -14523,11 +14524,11 @@ s64 bpf_kfunc_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn *
 	for (i = 0, slot = 0; i < nargs && slot < arg; i++)
 		slot += btf_arg_slots(btf_type_skip_modifiers(btf, args[i].type, NULL));
 	if (i >= nargs || slot != arg)
-		return 0;
+		return (struct arg_access_info) {};
 
 	t = btf_type_skip_modifiers(btf, args[i].type, NULL);
 	if (!btf_type_is_ptr(t))
-		return 0;
+		return (struct arg_access_info) {};
 
 	/* dynptr: fixed 16-byte on-stack representation */
 	if (is_kfunc_arg_dynptr(btf, &args[i])) {
@@ -14543,11 +14544,11 @@ s64 bpf_kfunc_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn *
 
 		if (size_reg <= MAX_BPF_FUNC_REG_ARGS &&
 		    (aux->const_reg_mask & BIT(size_reg))) {
-			size = (s64)aux->const_reg_vals[size_reg];
+			size = aux->const_reg_vals[size_reg];
 			goto out;
 		}
 		/* Unknown size: the read may extend anywhere up to the frame top. */
-		return S64_MIN;
+		return info;
 	}
 
 	/* fixed-size pointed-to type: resolve via BTF */
@@ -14557,15 +14558,17 @@ s64 bpf_kfunc_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn *
 		goto out;
 	}
 
-	return S64_MIN;
+	return info;
 out:
+	info.size = min_t(u64, size, U32_MAX);
 	/* KF_ITER_NEW kfuncs initialize the iterator state at arg 0 */
 	if (arg == 0 && meta.kfunc_flags & KF_ITER_NEW)
-		return -size;
+		info.may_read = false;
 	if (is_kfunc_arg_uninit(btf, &args[i]) &&
 	    (is_kfunc_arg_dynptr(btf, &args[i]) || env->allow_uninit_stack))
-		return -size;
-	return size;
+		info.may_read = false;
+	info.must_write = !info.may_read && info.size != U32_MAX;
+	return info;
 }
 
 /* check special kfuncs and return:
diff --git a/tools/testing/selftests/bpf/progs/verifier_kfunc_uninit.c b/tools/testing/selftests/bpf/progs/verifier_kfunc_uninit.c
index ff2fb36a0260..20ed71a54686 100644
--- a/tools/testing/selftests/bpf/progs/verifier_kfunc_uninit.c
+++ b/tools/testing/selftests/bpf/progs/verifier_kfunc_uninit.c
@@ -18,6 +18,9 @@ void __kfunc_btf_root(void)
 
 SEC("tc")
 __success __retval(10)
+__log_level(2)
+__msg("call bpf_kfunc_test_uninit_struct{{.*}}; def: fp0-8 fp0-16")
+__msg_unpriv("call bpf_kfunc_test_uninit_struct{{.*}}; use: fp0-8 fp0-16")
 __flag(BPF_F_TEST_STATE_FREQ)
 __caps_unpriv(CAP_BPF | CAP_NET_ADMIN)
 __prepare_priv
@@ -44,6 +47,9 @@ __naked void struct_poisoned_at_checkpoint(void)
 
 SEC("tc")
 __success __retval(0x2a2a2a2a)
+__log_level(2)
+__msg("call bpf_kfunc_test_uninit_mem{{.*}}; def: fp0-8")
+__msg_unpriv("call bpf_kfunc_test_uninit_mem{{.*}}; use: fp0-8")
 __flag(BPF_F_TEST_STATE_FREQ)
 __caps_unpriv(CAP_BPF | CAP_NET_ADMIN)
 __prepare_priv

-- 
2.53.0

[PATCH bpf-next v3 04/51] bpf: track may_write flags in liveness

[Eduard Zingerman]

Track stack slots that instructions may modify, in order to account
for such writes when constructing SCEV expressions.

Consume the may_write flag in arg_access_info separately from reads and
definite writes. Include partially covered slots and all candidate
offsets; imprecise writes mark whole candidate frames. Possible writes
do not kill liveness.

Merge analyzed instances as:

  may_write(dst)  |= may_write(src)
  must_write(dst) &= must_write(src)

For each ancestor frame f, summarize callee writes at the callsite:

  may_write(callsite, f) |= OR_{i in callee} may_write(i, f)

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 kernel/bpf/liveness.c                              | 207 ++++++++++++++-------
 .../selftests/bpf/progs/verifier_live_stack.c      |  38 ++--
 2 files changed, 168 insertions(+), 77 deletions(-)

diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index d4089a488784..e18d5d86c301 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -23,6 +23,7 @@ enum {
 	FM_MAY_READ,	/* stack slots that may be read by this instruction */
 	FM_MUST_WRITE,	/* stack slots written by this instruction */
 	FM_LIVE_BEFORE,	/* stack slots that may be read by this insn and its successors */
+	FM_MAY_WRITE,	/* stack slots that may be written by this instruction */
 	FM_MASK_CNT,
 };
 
@@ -262,6 +263,12 @@ static int mark_stack_write(struct func_instance *instance, u32 frame, u32 insn_
 	return mark_stack_range(instance, frame, insn_idx, FM_MUST_WRITE, lo, hi);
 }
 
+static int mark_stack_may_write(struct func_instance *instance, u32 frame, u32 insn_idx,
+				s32 lo, s32 hi)
+{
+	return mark_stack_range(instance, frame, insn_idx, FM_MAY_WRITE, lo, hi);
+}
+
 /*
  * Mark every half-slot of @frame as possibly read by @insn_idx. This widens
  * the masks to the program's stack budget: a full read recorded at a narrower
@@ -277,9 +284,16 @@ static int mark_stack_read_all(struct bpf_verifier_env *env, struct func_instanc
 			       env->stack_limit / BPF_HALF_REG_SIZE - 1);
 }
 
-/* Accumulate @src, a mask @src_words wide, into may_read of @frame at @insn_idx */
-static int mark_stack_read_mask(struct func_instance *instance, u32 frame, u32 insn_idx,
-				const unsigned long *src, u32 src_words)
+static int mark_stack_may_write_all(struct bpf_verifier_env *env, struct func_instance *instance,
+				    u32 frame, u32 insn_idx)
+{
+	return mark_stack_may_write(instance, frame, insn_idx, 0,
+				    env->stack_limit / BPF_HALF_REG_SIZE - 1);
+}
+
+/* Accumulate @src, a mask @src_words wide, into @kind mask of @frame at @insn_idx */
+static int mark_stack_mask(struct func_instance *instance, u32 frame, u32 insn_idx, u32 kind,
+			   const unsigned long *src, u32 src_words)
 {
 	u32 nbits = src_words * BITS_PER_LONG;
 	struct frame_masks *fm;
@@ -292,7 +306,7 @@ static int mark_stack_read_mask(struct func_instance *instance, u32 frame, u32 i
 	fm = widen_frame_masks(instance, frame, BITS_TO_LONGS(last + 1));
 	if (!fm)
 		return -ENOMEM;
-	dst = rel_mask(fm, relative_idx(instance, insn_idx), FM_MAY_READ);
+	dst = rel_mask(fm, relative_idx(instance, insn_idx), kind);
 	/* @src has no bits set past @last, hence none past @fm->words either */
 	src_words = min(src_words, fm->words);
 	for (w = 0; w < src_words; w++)
@@ -300,6 +314,12 @@ static int mark_stack_read_mask(struct func_instance *instance, u32 frame, u32 i
 	return 0;
 }
 
+static int mark_stack_read_mask(struct func_instance *instance, u32 frame, u32 insn_idx,
+				const unsigned long *src, u32 src_words)
+{
+	return mark_stack_mask(instance, frame, insn_idx, FM_MAY_READ, src, src_words);
+}
+
 int bpf_jmp_offset(struct bpf_insn *insn)
 {
 	u8 code = insn->code;
@@ -624,16 +644,41 @@ static char *fmt_spis_mask(struct bpf_verifier_env *env, int frame, bool first,
 	return env->tmp_str_buf;
 }
 
+/* Print mask @kind of the instruction at relative index @i for every frame, if any bit is set. */
+static bool print_mask(struct bpf_verifier_env *env, struct func_instance *instance, int i,
+		       const char *name, u32 kind)
+{
+	struct frame_masks *fm;
+	bool printed = false;
+	unsigned long *mask;
+	int frame;
+	u64 pos;
+
+	pos = env->log.end_pos;
+	verbose(env, "%s", name);
+	for (frame = instance->depth; frame >= 0; --frame) {
+		fm = instance->frames[frame];
+		if (!fm)
+			continue;
+		mask = rel_mask(fm, i, kind);
+		if (bitmap_empty(mask, frame_mask_bits(fm)))
+			continue;
+		verbose(env, "%s", fmt_spis_mask(env, frame, !printed, mask, fm->words));
+		printed = true;
+	}
+	if (!printed)
+		bpf_vlog_reset(&env->log, pos);
+	return printed;
+}
+
 static void print_instance(struct bpf_verifier_env *env, struct func_instance *instance)
 {
 	int start = env->subprog_info[instance->subprog].start;
 	struct bpf_insn *insns = env->prog->insnsi;
-	struct frame_masks *fm;
-	unsigned long *mask;
 	int len = instance->insn_cnt;
-	int insn_idx, frame, i;
-	bool has_use, has_def;
 	u64 pos, insn_pos;
+	int insn_idx, i;
+	bool printed;
 
 	if (!(env->log.level & BPF_LOG_LEVEL2))
 		return;
@@ -642,41 +687,17 @@ static void print_instance(struct bpf_verifier_env *env, struct func_instance *i
 	verbose(env, "%s:\n", fmt_instance(env, instance));
 	for (i = 0; i < len; i++) {
 		insn_idx = start + i;
-		has_use = false;
-		has_def = false;
 		pos = env->log.end_pos;
 		verbose(env, "%3d: ", insn_idx);
 		bpf_verbose_insn(env, &insns[insn_idx]);
 		insn_pos = env->log.end_pos;
 		verbose(env, "%*c;", bpf_vlog_alignment(insn_pos - pos), ' ');
-		pos = env->log.end_pos;
-		verbose(env, " use: ");
-		for (frame = instance->depth; frame >= 0; --frame) {
-			fm = instance->frames[frame];
-			if (!fm)
-				continue;
-			mask = rel_mask(fm, i, FM_MAY_READ);
-			if (bitmap_empty(mask, frame_mask_bits(fm)))
-				continue;
-			verbose(env, "%s", fmt_spis_mask(env, frame, !has_use, mask, fm->words));
-			has_use = true;
-		}
-		if (!has_use)
-			bpf_vlog_reset(&env->log, pos);
-		pos = env->log.end_pos;
-		verbose(env, " def: ");
-		for (frame = instance->depth; frame >= 0; --frame) {
-			fm = instance->frames[frame];
-			if (!fm)
-				continue;
-			mask = rel_mask(fm, i, FM_MUST_WRITE);
-			if (bitmap_empty(mask, frame_mask_bits(fm)))
-				continue;
-			verbose(env, "%s", fmt_spis_mask(env, frame, !has_def, mask, fm->words));
-			has_def = true;
-		}
-		if (!has_def)
-			bpf_vlog_reset(&env->log, has_use ? pos : insn_pos);
+		printed = false;
+		printed |= print_mask(env, instance, i, " use: ", FM_MAY_READ);
+		printed |= print_mask(env, instance, i, " def: ", FM_MUST_WRITE);
+		printed |= print_mask(env, instance, i, " may_def: ", FM_MAY_WRITE);
+		if (!printed)
+			bpf_vlog_reset(&env->log, insn_pos);
 		verbose(env, "\n");
 		if (bpf_is_ldimm64(&insns[insn_idx]))
 			i++;
@@ -1437,12 +1458,14 @@ static void arg_track_xfer(struct bpf_verifier_env *env, struct bpf_insn *insn,
 }
 
 /*
- * Record reads for every touched half-slot. A definite
+ * Record possible reads and writes for every touched half-slot. A definite
  * write requires full coverage, a known size, and a single possible offset.
  */
-static int record_stack_access_off(struct func_instance *instance, s64 fp_off,
-				   struct arg_access_info info, u32 frame, u32 insn_idx)
+static int record_stack_access_off(struct func_instance *instance, const struct arg_track *arg,
+				   s64 off_idx, struct arg_access_info info,
+				   u32 frame, u32 insn_idx)
 {
+	s64 fp_off = arg->off[off_idx];
 	s32 slot_hi, slot_lo;
 	int err;
 
@@ -1454,20 +1477,25 @@ static int record_stack_access_off(struct func_instance *instance, s64 fp_off,
 		 */
 		return 0;
 
-	if (info.may_read && info.size == U32_MAX) {
-		/* helper/kfunc read unknown amount of bytes from fp_off until fp+0 */
-		slot_hi = (-fp_off - 1) / STACK_SLOT_SZ;
-		return mark_stack_read(instance, frame, insn_idx, 0, slot_hi);
-	}
+	/*
+	 * Read and may_write marks include partially covered slots.
+	 * An unknown access size may reach from fp_off to the frame top.
+	 */
+	slot_hi = (-fp_off - 1) / STACK_SLOT_SZ;
+	slot_lo = info.size == U32_MAX
+		  ? 0
+		  : max_t(s32, (-fp_off - info.size) / STACK_SLOT_SZ, 0);
 	if (info.may_read) {
-		/* Mark any touched slot as use */
-		slot_hi = (-fp_off - 1) / STACK_SLOT_SZ;
-		slot_lo = max_t(s32, (-fp_off - info.size) / STACK_SLOT_SZ, 0);
 		err = mark_stack_read(instance, frame, insn_idx, slot_lo, slot_hi);
 		if (err)
 			return err;
 	}
-	if (info.must_write && info.size != U32_MAX) {
+	if (info.may_write) {
+		err = mark_stack_may_write(instance, frame, insn_idx, slot_lo, slot_hi);
+		if (err)
+			return err;
+	}
+	if (info.must_write && info.size != U32_MAX && arg->off_cnt == 1) {
 		/* Mark only fully covered slots as def */
 		slot_hi = (-fp_off) / STACK_SLOT_SZ - 1;
 		slot_lo = max_t(s32, (-fp_off - info.size + STACK_SLOT_SZ - 1) / STACK_SLOT_SZ, 0);
@@ -1492,26 +1520,24 @@ static int record_stack_access(struct bpf_verifier_env *env,
 			if (err)
 				return err;
 		}
+		if (info.may_write) {
+			err = mark_stack_may_write_all(env, instance, frame, insn_idx);
+			if (err)
+				return err;
+		}
 		return 0;
 	}
-	if (info.size != U32_MAX && info.must_write && arg->off_cnt != 1)
-		/* multi-offset write cannot set stack_def */
-		return 0;
-
 	for (i = 0; i < arg->off_cnt; i++) {
-		err = record_stack_access_off(instance, arg->off[i], info, frame, insn_idx);
+		err = record_stack_access_off(instance, arg, i, info, frame, insn_idx);
 		if (err)
 			return err;
 	}
 	return 0;
 }
 
-/*
- * When a pointer is ARG_IMPRECISE, conservatively mark every frame in
- * the bitmask as fully used.
- */
+/* Record possible effects on every candidate frame of an imprecise pointer. */
 static int record_imprecise(struct bpf_verifier_env *env, struct func_instance *instance,
-			    u32 mask, u32 insn_idx)
+			    struct arg_access_info info, u32 mask, u32 insn_idx)
 {
 	int depth = instance->depth;
 	int f, err;
@@ -1520,9 +1546,16 @@ static int record_imprecise(struct bpf_verifier_env *env, struct func_instance *
 		if (!(mask & 1))
 			continue;
 		if (f <= depth) {
-			err = mark_stack_read_all(env, instance, f, insn_idx);
-			if (err)
-				return err;
+			if (info.may_read) {
+				err = mark_stack_read_all(env, instance, f, insn_idx);
+				if (err)
+					return err;
+			}
+			if (info.may_write) {
+				err = mark_stack_may_write_all(env, instance, f, insn_idx);
+				if (err)
+					return err;
+			}
 		}
 	}
 	return 0;
@@ -1599,7 +1632,7 @@ static int record_load_store_access(struct bpf_verifier_env *env,
 	if (ptr->frame >= 0 && ptr->frame <= depth)
 		return record_stack_access(env, instance, ptr, info, ptr->frame, insn_idx);
 	if (ptr->frame == ARG_IMPRECISE)
-		return record_imprecise(env, instance, ptr->mask, insn_idx);
+		return record_imprecise(env, instance, info, ptr->mask, insn_idx);
 	/* ARG_NONE: not derived from any frame pointer, skip */
 	return 0;
 }
@@ -1627,6 +1660,9 @@ static int record_arg_access(struct bpf_verifier_env *env,
 			err = mark_stack_read_all(env, instance, f, insn_idx);
 			if (err)
 				return err;
+			err = mark_stack_may_write_all(env, instance, f, insn_idx);
+			if (err)
+				return err;
 		}
 		return 0;
 	}
@@ -1636,7 +1672,7 @@ static int record_arg_access(struct bpf_verifier_env *env,
 	if (frame >= 0 && frame <= depth)
 		err = record_stack_access(env, instance, at, info, frame, insn_idx);
 	else if (frame == ARG_IMPRECISE)
-		err = record_imprecise(env, instance, at->mask, insn_idx);
+		err = record_imprecise(env, instance, info, at->mask, insn_idx);
 	return err;
 }
 
@@ -1996,6 +2032,7 @@ static bool has_fp_args(struct arg_track *args)
 /*
  * Merge a freshly analyzed instance into the original.
  * may_read: union (any pass might read the slot).
+ * may_write: union (slots written on ANY pass).
  * must_write: intersection (only slots written on ALL passes are guaranteed).
  * live_before is recomputed by a subsequent update_instance() on @dst.
  *
@@ -2034,18 +2071,50 @@ static int merge_instances(struct func_instance *dst, struct func_instance *src)
 		for (i = 0; i < dst->insn_cnt; i++) {
 			unsigned long *dst_read = rel_mask(d, i, FM_MAY_READ);
 			unsigned long *dst_write = rel_mask(d, i, FM_MUST_WRITE);
+			unsigned long *dst_may_write = rel_mask(d, i, FM_MAY_WRITE);
 			unsigned long *src_read = rel_mask(s, i, FM_MAY_READ);
 			unsigned long *src_write = rel_mask(s, i, FM_MUST_WRITE);
+			unsigned long *src_may_write = rel_mask(s, i, FM_MAY_WRITE);
 
 			for (w = 0; w < d->words; w++) {
 				dst_read[w] |= w < s->words ? src_read[w] : 0;
 				dst_write[w] &= w < s->words ? src_write[w] : 0;
+				dst_may_write[w] |= w < s->words ? src_may_write[w] : 0;
 			}
 		}
 	}
 	return 0;
 }
 
+/*
+ * Fold a fully analyzed callee instance writes to upper frames as
+ * may_write marks at callsite in caller's frames.
+ */
+static int merge_may_write(struct func_instance *caller, struct func_instance *callee)
+{
+	DECLARE_BITMAP(acc, FRAME_HALF_SPIS);
+	u32 call_idx = callee->callsite;
+	struct frame_masks *fm;
+	u32 f, i, nbits;
+	int err;
+
+	for (f = 0; f < callee->depth; f++) {
+		fm = callee->frames[f];
+		if (!fm)
+			continue;
+		nbits = frame_mask_bits(fm);
+		bitmap_zero(acc, nbits);
+		for (i = 0; i < callee->insn_cnt; i++)
+			bitmap_or(acc, acc, rel_mask(fm, i, FM_MAY_WRITE), nbits);
+		if (bitmap_empty(acc, nbits))
+			continue;
+		err = mark_stack_mask(caller, f, call_idx, FM_MAY_WRITE, acc, fm->words);
+		if (err)
+			return err;
+	}
+	return 0;
+}
+
 static struct func_instance *fresh_instance(struct func_instance *src)
 {
 	struct func_instance *f;
@@ -2161,6 +2230,9 @@ static int analyze_subprog(struct bpf_verifier_env *env,
 					err = mark_stack_read_all(env, instance, f, idx);
 					if (err)
 						goto out_free;
+					err = mark_stack_may_write_all(env, instance, f, idx);
+					if (err)
+						goto out_free;
 				}
 				continue;
 			}
@@ -2213,6 +2285,11 @@ static int analyze_subprog(struct bpf_verifier_env *env,
 					goto out_free;
 			}
 		}
+
+		/* Summarize callee's writes to ancestor frames onto the callsite */
+		err = merge_may_write(instance, callee_instance);
+		if (err)
+			goto out_free;
 	}
 
 	if (prev_instance) {
diff --git a/tools/testing/selftests/bpf/progs/verifier_live_stack.c b/tools/testing/selftests/bpf/progs/verifier_live_stack.c
index 16b2b1e57534..3ba21430bb75 100644
--- a/tools/testing/selftests/bpf/progs/verifier_live_stack.c
+++ b/tools/testing/selftests/bpf/progs/verifier_live_stack.c
@@ -61,7 +61,11 @@ __naked void read_write_join(void)
 SEC("socket")
 __log_level(2)
 __msg("stack use/def subprog#0 must_write_not_same_slot (d0,cs0):")
-__msg("6: (7b) *(u64 *)(r2 +0) = r0{{$}}")
+/*
+ * 'r2 += r1' adds a scalar, so the offset is lost (off_cnt == 0): no def,
+ * but the write conservatively marks the whole frame as may_def.
+ */
+__msg("6: (7b) *(u64 *)(r2 +0) = r0         ; may_def: fp0-8..-{{(512|2048)}}")
 __msg("Live regs before insn:")
 __naked void must_write_not_same_slot(void)
 {
@@ -106,9 +110,9 @@ __naked void must_write_not_same_type(void)
 
 SEC("socket")
 __log_level(2)
-/* Callee writes fp[0]-8: stack_use at call site has slots 0,1 live */
+/* Callee writes fp[0]-8: the def is summarized as may_def at the call site */
 __msg("stack use/def subprog#0 caller_stack_write (d0,cs0):")
-__msg("2: (85) call pc+1{{$}}")
+__msg("2: (85) call pc+1                    ; may_def: fp0-8")
 __msg("stack use/def subprog#1 write_first_param (d1,cs2):")
 __msg("4: (7a) *(u64 *)(r1 +0) = 7          ; def: fp0-8")
 __naked void caller_stack_write(void)
@@ -804,9 +808,9 @@ void __kfunc_btf_root(void)
  */
 SEC("socket")
 __success __log_level(2)
-__msg(" 6: (85) call bpf_iter_num_new{{.*}}          ; def: fp0-24{{$}}")
-__msg(" 9: (85) call bpf_iter_num_next{{.*}}         ; use: fp0-24{{$}}")
-__msg("14: (85) call bpf_iter_num_destroy{{.*}}      ; use: fp0-24{{$}}")
+__msg(" 6: (85) call bpf_iter_num_new{{.*}}          ; def: fp0-24 may_def: fp0-24{{$}}")
+__msg(" 9: (85) call bpf_iter_num_next{{.*}}         ; use: fp0-24 may_def: fp0-24{{$}}")
+__msg("14: (85) call bpf_iter_num_destroy{{.*}}      ; use: fp0-24 may_def: fp0-24{{$}}")
 __naked void kfunc_iter_stack_liveness(void)
 {
 	asm volatile (
@@ -1008,7 +1012,8 @@ __naked void four_byte_read_upper_half(void)
 SEC("socket")
 __log_level(2)
 __msg("0: (7a) *(u64 *)(r10 -8) = 0         ; def: fp0-8")
-__msg("1: (6a) *(u16 *)(r10 -4) = 0{{$}}")
+/* 2-byte write only partially covers the upper half: may_def, but no def. */
+__msg("1: (6a) *(u16 *)(r10 -4) = 0         ; may_def: fp0-4h")
 __msg("2: (61) r0 = *(u32 *)(r10 -4)        ; use: fp0-4h")
 __naked void two_byte_write_no_kill(void)
 {
@@ -1355,9 +1360,12 @@ __naked void fp_spill_loses_precision_kills_liveness(void)
  */
 SEC("socket")
 __log_level(2)
-/* fp-8 live at call (callee conditionally writes → slot not killed) */
+/*
+ * fp-8 live at call: callee conditionally writes it, so the slot is not killed
+ * (no def), but the conditional write surfaces as may_def at the call site.
+ */
 __msg("1: (7b) *(u64 *)(r10 -8) = r1        ; def: fp0-8")
-__msg("4: (85) call pc+2{{$}}")
+__msg("4: (85) call pc+2                    ; may_def: fp0-8")
 __msg("5: (79) r0 = *(u64 *)(r10 -8)        ; use: fp0-8")
 __naked void conditional_stx_in_subprog(void)
 {
@@ -2386,7 +2394,12 @@ __msg("subprog#2 write_first_read_second:")
 __msg("17: (7a) *(u64 *)(r1 +0) = 42{{$}}")
 __msg("18: (79) r0 = *(u64 *)(r2 +0) // r1=fp0-8 r2=fp0-16{{$}}")
 __msg("stack use/def subprog#2 write_first_read_second (d2,cs15):")
-__msg("17: (7a) *(u64 *)(r1 +0) = 42{{$}}")
+/*
+ * Shared across two callsites with swapped args (r1 is fp-8 on one pass,
+ * fp-16 on the other): must_write intersects to empty (no def), may_write
+ * unions to both slots.
+ */
+__msg("17: (7a) *(u64 *)(r1 +0) = 42         ; may_def: fp0-8 fp0-16")
 __msg("18: (79) r0 = *(u64 *)(r2 +0)         ; use: fp0-8 fp0-16")
 __naked void shared_instance_must_write_overwrite(void)
 {
@@ -2848,8 +2861,9 @@ static __used __naked void imprecise_dst_spill_join_sub(void)
 SEC("socket")
 __log_level(2)
 __msg("0: (79) r0 = *(u64 *)(r10 -8)        ; use: fp0-8")
-__msg("1: (73) *(u8 *)(r10 -1) = r0{{$}}")
-__msg("2: (6b) *(u16 *)(r10 -4) = r0{{$}}")
+/* narrow stores define nothing, but they may write the half-slot they touch */
+__msg("1: (73) *(u8 *)(r10 -1) = r0         ; may_def: fp0-4h")
+__msg("2: (6b) *(u16 *)(r10 -4) = r0        ; may_def: fp0-4h")
 __msg("3: (79) r0 = *(u64 *)(r10 -8)        ; use: fp0-8")
 __naked void narrow_store_defines_nothing(void)
 {

-- 
2.53.0

[PATCH bpf-next v3 05/51] bpf: summarize may write stack slots in insn_aux_data

[Eduard Zingerman]

SCEV needs may-write information to invalidate stack expressions after
indirect writes. Liveness tracks this per function instance,
but SCEV is not callchain-sensitive and needs a summary across
calling contexts.

For each instruction, union the current-frame may_write masks from all
analyzed instances. Represent 8-byte slots as a summary of constituent
halves. Store the summary in insn_aux_data and expose it via a helper.

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 include/linux/bpf_verifier.h |  6 ++++++
 kernel/bpf/liveness.c        | 41 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 47 insertions(+)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index eafe79929dfa..4cb36ee3b6f7 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -662,6 +662,11 @@ struct bpf_insn_aux_data {
 	};
 	struct btf_struct_meta *kptr_struct_meta;
 	u64 map_key_state; /* constant (32 bit) key tracking for maps */
+	/*
+	 * Per-instruction summary of stack slots in the current frame
+	 * that this instruction may write to.
+	 */
+	DECLARE_BITMAP(may_write_mask, MAX_BPF_STACK_SLOTS);
 	int ctx_field_size; /* the ctx field size for load insn, maybe 0 */
 	u32 seen; /* this insn was processed by the verifier at env->pass_cnt */
 	bool nospec; /* do not execute this instruction speculatively */
@@ -1742,6 +1747,7 @@ int bpf_compute_subprog_arg_access(struct bpf_verifier_env *env);
 int bpf_stack_liveness_init(struct bpf_verifier_env *env);
 void bpf_stack_liveness_free(struct bpf_verifier_env *env);
 int bpf_live_stack_query_init(struct bpf_verifier_env *env, struct bpf_verifier_state *st);
+const unsigned long *bpf_may_write_mask(struct bpf_verifier_env *env, u32 insn_idx);
 bool bpf_stack_slot_alive(struct bpf_verifier_env *env, u32 frameno, u32 spi);
 int bpf_compute_live_registers(struct bpf_verifier_env *env);
 
diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index e18d5d86c301..ec9c8beef556 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -718,6 +718,45 @@ static int cmp_instances(const void *pa, const void *pb)
 	return 0;
 }
 
+/* OR the 8-byte slots touched by a half-slot (4-byte) mask into @slots. */
+static void half_spis_to_slots(unsigned long *slots, const unsigned long *mask, u32 nbits)
+{
+	u32 slot;
+
+	for (slot = 0; slot < MAX_BPF_STACK_SLOTS && slot * 2 + 1 < nbits; slot++)
+		if (test_bit(slot * 2, mask) || test_bit(slot * 2 + 1, mask))
+			__set_bit(slot, slots);
+}
+
+/*
+ * Precompute, for each instruction, the OR of may_write masks over its top
+ * frame across all func_instances reaching it, stash it in the insn_aux_data.
+ */
+static void compute_may_write_masks(struct bpf_verifier_env *env)
+{
+	struct bpf_insn_aux_data *aux = env->insn_aux_data;
+	struct bpf_liveness *liveness = env->liveness;
+	struct func_instance *instance;
+	struct frame_masks *fm;
+	u32 nbits;
+	int bkt, i;
+
+	hash_for_each(liveness->func_instances, bkt, instance, hl_node) {
+		fm = instance->frames[instance->depth];
+		if (!fm)
+			continue;
+		nbits = frame_mask_bits(fm);
+		for (i = 0; i < instance->insn_cnt; i++)
+			half_spis_to_slots(aux[instance->subprog_start + i].may_write_mask,
+					   rel_mask(fm, i, FM_MAY_WRITE), nbits);
+	}
+}
+
+const unsigned long *bpf_may_write_mask(struct bpf_verifier_env *env, u32 insn_idx)
+{
+	return env->insn_aux_data[insn_idx].may_write_mask;
+}
+
 /* print use/def slots for all instances ordered by callsite first, then by depth */
 static int print_instances(struct bpf_verifier_env *env)
 {
@@ -2352,6 +2391,8 @@ int bpf_compute_subprog_arg_access(struct bpf_verifier_env *env)
 			goto out;
 	}
 
+	compute_may_write_masks(env);
+
 	if (env->log.level & BPF_LOG_LEVEL2)
 		err = print_instances(env);
 

-- 
2.53.0

[PATCH bpf-next v3 06/51] bpf: summarize live stack slots in insn_aux_data

[Eduard Zingerman]

SCEV needs live stack information in order to skip tracking dead
slots. Liveness tracks this per function instance, but SCEV is not
callchain-sensitive and needs a summary across calling contexts.

For each instruction, union the current-frame liveness masks from all
analyzed function instances. Represent 8-byte slots as a summary of
constituent halves. Store the summary in insn_aux_data.

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 include/linux/bpf_verifier.h |  1 +
 kernel/bpf/liveness.c        | 10 +++++++---
 2 files changed, 8 insertions(+), 3 deletions(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 4cb36ee3b6f7..5b615083ffa9 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -667,6 +667,7 @@ struct bpf_insn_aux_data {
 	 * that this instruction may write to.
 	 */
 	DECLARE_BITMAP(may_write_mask, MAX_BPF_STACK_SLOTS);
+	DECLARE_BITMAP(live_stack_before, MAX_BPF_STACK_SLOTS);
 	int ctx_field_size; /* the ctx field size for load insn, maybe 0 */
 	u32 seen; /* this insn was processed by the verifier at env->pass_cnt */
 	bool nospec; /* do not execute this instruction speculatively */
diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index ec9c8beef556..004c1f48a065 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -729,8 +729,9 @@ static void half_spis_to_slots(unsigned long *slots, const unsigned long *mask,
 }
 
 /*
- * Precompute, for each instruction, the OR of may_write masks over its top
- * frame across all func_instances reaching it, stash it in the insn_aux_data.
+ * Precompute, for each instruction, the OR of may_write and live_before masks
+ * over its top frame across all func_instances reaching it, stash them in the
+ * insn_aux_data.
  */
 static void compute_may_write_masks(struct bpf_verifier_env *env)
 {
@@ -746,9 +747,12 @@ static void compute_may_write_masks(struct bpf_verifier_env *env)
 		if (!fm)
 			continue;
 		nbits = frame_mask_bits(fm);
-		for (i = 0; i < instance->insn_cnt; i++)
+		for (i = 0; i < instance->insn_cnt; i++) {
 			half_spis_to_slots(aux[instance->subprog_start + i].may_write_mask,
 					   rel_mask(fm, i, FM_MAY_WRITE), nbits);
+			half_spis_to_slots(aux[instance->subprog_start + i].live_stack_before,
+					   rel_mask(fm, i, FM_LIVE_BEFORE), nbits);
+		}
 	}
 }
 

-- 
2.53.0

[PATCH bpf-next v3 07/51] bpf: summarize regs that may hold a frame pointer in insn_aux_data

[Eduard Zingerman]

SCEV needs to know which registers might be stack pointers at a
particular instruction. This information is used to avoid widening
registers if that would hinder further verification.
See the patch [1] for additional information.

liveness.c:compute_subprog_args() already tracks this information.
This commit modifies it to save the information in insn_aux_data for
further usage.

[1] "bpf: avoid widening registers that hinder exact stack-slot tracking"

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 include/linux/bpf_verifier.h |  1 +
 kernel/bpf/liveness.c        | 13 +++++++++++++
 2 files changed, 14 insertions(+)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 5b615083ffa9..d31c1a500958 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -668,6 +668,7 @@ struct bpf_insn_aux_data {
 	 */
 	DECLARE_BITMAP(may_write_mask, MAX_BPF_STACK_SLOTS);
 	DECLARE_BITMAP(live_stack_before, MAX_BPF_STACK_SLOTS);
+	u16 stack_ptrs; /* bitmask of regs that may hold a frame pointer here (arg_track) */
 	int ctx_field_size; /* the ctx field size for load insn, maybe 0 */
 	u32 seen; /* this insn was processed by the verifier at env->pass_cnt */
 	bool nospec; /* do not execute this instruction speculatively */
diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index 004c1f48a065..b9b76684906e 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -1896,6 +1896,17 @@ static void print_subprog_arg_access(struct bpf_verifier_env *env,
 	}
 }
 
+static void record_stack_ptrs(struct bpf_verifier_env *env, int idx, struct arg_track *at_in)
+{
+	u16 r, mask = 0;
+
+	for (r = 0; r < MAX_BPF_REG; r++)
+		if (arg_is_fp(&at_in[r]))
+			mask |= BIT(r);
+
+	env->insn_aux_data[idx].stack_ptrs |= mask;
+}
+
 /*
  * Compute arg tracking dataflow for a single subprog.
  * Runs forward fixed-point with arg_track_xfer(), then records
@@ -2045,6 +2056,8 @@ static int compute_subprog_args(struct bpf_verifier_env *env,
 			snap->slots = nslots;
 			memcpy(snap->at, &at_stack_in[(size_t)i * nslots], nslots * sizeof(*snap->at));
 		}
+
+		record_stack_ptrs(env, idx, at_in[i]);
 	}
 
 	info->at_in = at_in;

-- 
2.53.0

[PATCH bpf-next v3 08/51] bpf: record write effects for atomic operations in liveness.c

[Eduard Zingerman]

Atomic read-modify-write instructions are recorded as reads only,
omitting their stack write effects from the may_write masks needed
by SCEV.

Record both read and write accesses for atomic RMW operations. Keep
LOAD_ACQ read-only and STORE_REL write-only. Share the precise/imprecise
stack-access dispatch so both accesses use the same address handling.

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 kernel/bpf/liveness.c | 20 ++++++++++++++------
 1 file changed, 14 insertions(+), 6 deletions(-)

diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index b9b76684906e..ce74e3f16eef 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -1634,14 +1634,22 @@ static int record_load_store_access(struct bpf_verifier_env *env,
 		break;
 	case BPF_STX:
 		if (BPF_MODE(insn->code) == BPF_ATOMIC) {
-			if (insn->imm == BPF_STORE_REL)
-				write = true;
-			else
-				read = true;
-			if (insn->imm == BPF_LOAD_ACQ)
+			switch (insn->imm) {
+			case BPF_LOAD_ACQ:
 				ptr = &at[insn->src_reg];
-			else
+				read = true;
+				break;
+			case BPF_STORE_REL:
 				ptr = &at[insn->dst_reg];
+				write = true;
+				break;
+			default:
+				/* ADD/AND/OR/XOR(+FETCH), XCHG, CMPXCHG */
+				ptr = &at[insn->dst_reg];
+				read = true;
+				write = true;
+				break;
+			}
 		} else {
 			ptr = &at[insn->dst_reg];
 			write = true;

-- 
2.53.0

[PATCH bpf-next v3 09/51] bpf: fix arg_track_xfer() for atomic stack spills and fills

[Eduard Zingerman]

Track spilled stack pointers through 64-bit atomic accesses:
- BPF_STORE_REL spills the source register.
- BPF_LOAD_ACQ restores the spilled value into dst_reg.
- BPF_XCHG and BPF_{ADD,AND,OR,XOR} | BPF_FETCH restore it into src_reg.
- BPF_CMPXCHG restores it into R0.

This prevents missing stack accesses through pointers recovered by these
instructions, matching the main verifier's spill/fill behavior.

Fixes: bf0c571f7feb ("bpf: introduce forward arg-tracking dataflow analysis")
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 kernel/bpf/liveness.c | 91 ++++++++++++++++++++++++++++-----------------------
 1 file changed, 50 insertions(+), 41 deletions(-)

diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index ce74e3f16eef..52c39bbee6b1 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -1357,6 +1357,29 @@ static bool can_be_local_fp(int depth, int regno, struct arg_track *at)
 	       (at->frame == ARG_IMPRECISE && (at->mask & BIT(depth)));
 }
 
+/*
+ * Model 8-byte reads through @reg, including load-acquire
+ * and the old value fetched by atomic operations.
+ */
+static struct arg_track arg_track_deref(struct bpf_verifier_env *env, struct bpf_insn *insn,
+					struct arg_track *at_out, int reg,
+					struct arg_track *at_stack_out, u32 nslots,
+					int depth, u32 *callsites)
+{
+	struct arg_track *ptr = &at_out[reg];
+
+	if (can_be_local_fp(depth, reg, ptr))
+		return fill_from_stack(insn, at_out, reg, at_stack_out, nslots, depth);
+	if (ptr->frame >= 0 && ptr->frame < depth) {
+		struct spill_snapshot *parent = env->callsite_at_stack[callsites[ptr->frame]];
+
+		return fill_from_stack(insn, at_out, reg, parent->at, parent->slots, ptr->frame);
+	}
+	if (ptr->frame == ARG_IMPRECISE && !(ptr->mask & BIT(depth)) && ptr->mask)
+		return *ptr;
+	return (struct arg_track){ .frame = ARG_NONE };
+}
+
 /*
  * Pure dataflow transfer function for arg_track state.
  * Updates at_out[] based on how the instruction modifies registers.
@@ -1438,56 +1461,42 @@ static void arg_track_xfer(struct bpf_verifier_env *env, struct bpf_insn *insn,
 		for (r = BPF_REG_0; r <= BPF_REG_5; r++)
 			at_out[r] = none;
 	} else if (class == BPF_LDX) {
-		u32 sz = bpf_size_to_bytes(BPF_SIZE(insn->code));
-		bool src_is_local_fp = can_be_local_fp(depth, insn->src_reg, src);
-
-		/*
-		 * Reload from callee stack: if src is current-frame FP-derived
-		 * and the load is an 8-byte BPF_MEM, try to restore the spill
-		 * identity.  For imprecise sources fill_from_stack() returns
-		 * ARG_IMPRECISE (off_cnt == 0).
-		 */
-		if (src_is_local_fp && BPF_MODE(insn->code) == BPF_MEM && sz == 8) {
-			*dst = fill_from_stack(insn, at_out, insn->src_reg, at_stack_out, nslots,
-					       depth);
-		} else if (src->frame >= 0 && src->frame < depth &&
-			   BPF_MODE(insn->code) == BPF_MEM && sz == 8) {
-			struct spill_snapshot *parent =
-				env->callsite_at_stack[callsites[src->frame]];
-
-			*dst = fill_from_stack(insn, at_out, insn->src_reg,
-					       parent->at, parent->slots, src->frame);
-		} else if (src->frame == ARG_IMPRECISE &&
-			   !(src->mask & BIT(depth)) && src->mask &&
-			   BPF_MODE(insn->code) == BPF_MEM && sz == 8) {
-			/*
-			 * Imprecise src with only parent-frame bits:
-			 * conservative fallback.
-			 */
-			*dst = *src;
-		} else {
+		/* Only an 8-byte BPF_MEM load can be a fill, see arg_track_deref(). */
+		if (BPF_MODE(insn->code) == BPF_MEM && BPF_SIZE(insn->code) == BPF_DW)
+			*dst = arg_track_deref(env, insn, at_out, insn->src_reg,
+					       at_stack_out, nslots, depth, callsites);
+		else
 			*dst = none;
-		}
 	} else if (class == BPF_LD && BPF_MODE(insn->code) == BPF_IMM) {
 		*dst = none;
 	} else if (class == BPF_STX) {
 		u32 sz = bpf_size_to_bytes(BPF_SIZE(insn->code));
-		bool dst_is_local_fp;
-
-		/* Track spills to current-frame FP-derived callee stack */
-		dst_is_local_fp = can_be_local_fp(depth, insn->dst_reg, dst);
-		if (dst_is_local_fp && BPF_MODE(insn->code) == BPF_MEM)
-			spill_to_stack(insn, at_out, insn->dst_reg,
-				       at_stack_out, nslots, src, sz);
+		bool dst_is_local_fp = can_be_local_fp(depth, insn->dst_reg, dst);
+		u8 mode = BPF_MODE(insn->code);
+
+		if (mode == BPF_MEM || (mode == BPF_ATOMIC && insn->imm == BPF_STORE_REL)) {
+			if (dst_is_local_fp)
+				spill_to_stack(insn, at_out, insn->dst_reg,
+					       at_stack_out, nslots, src, sz);
+		} else if (mode == BPF_ATOMIC && insn->imm == BPF_LOAD_ACQ) {
+			if (sz == 8)
+				*dst = arg_track_deref(env, insn, at_out, insn->src_reg,
+						       at_stack_out, nslots, depth, callsites);
+			else
+				*dst = none;
+		} else if (mode == BPF_ATOMIC) {
+			/* BPF_XCHG, BPF_CMPXCHG or BPF_{ADD,AND,OR,XOR} | [BPF_FETCH] */
+			struct arg_track old = none;
 
-		if (BPF_MODE(insn->code) == BPF_ATOMIC) {
-			if (dst_is_local_fp && insn->imm != BPF_LOAD_ACQ)
+			r = bpf_atomic_load_reg(insn);
+			if (r >= 0 && sz == 8)
+				old = arg_track_deref(env, insn, at_out, insn->dst_reg,
+						      at_stack_out, nslots, depth, callsites);
+			if (dst_is_local_fp)
 				clear_stack_for_all_offs(insn, at_out, insn->dst_reg,
 							 at_stack_out, nslots, sz);
-
-			r = bpf_atomic_load_reg(insn);
 			if (r >= 0)
-				at_out[r] = none;
+				at_out[r] = old;
 		}
 	} else if (class == BPF_ST && BPF_MODE(insn->code) == BPF_MEM) {
 		u32 sz = bpf_size_to_bytes(BPF_SIZE(insn->code));

-- 
2.53.0

[PATCH bpf-next v3 10/51] bpf: derive global subprog stack liveness from BTF signatures

[Eduard Zingerman]

Stack liveness currently analyzes global callees in the context of their
callers. The resulting read/write marks describe only the current body,
but freplace can replace it after the caller has been verified.

A no-op callee taking a stack pointer can be replaced with one writing
through that pointer, leaving the call's may_write marks incomplete.
SCEV could then preserve a stack counter recurrence across a call that
resets it, invalidating a loop termination proof.

Instead, derive global call effects from BTF argument types instead,
same is it done on the main verification pass.

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 include/linux/bpf_verifier.h |  3 ++
 kernel/bpf/liveness.c        | 28 ++++++++++++++-----
 kernel/bpf/verifier.c        | 66 ++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 90 insertions(+), 7 deletions(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index d31c1a500958..05e6f091ad94 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1744,6 +1744,9 @@ bpf_helper_stack_access_bytes(struct bpf_verifier_env *env,
 struct arg_access_info
 bpf_kfunc_stack_access_bytes(struct bpf_verifier_env *env,
 			     struct bpf_insn *insn, int arg, int insn_idx);
+struct arg_access_info
+bpf_global_subprog_stack_access_bytes(struct bpf_verifier_env *env,
+				      struct bpf_insn *insn, int arg, int insn_idx);
 int bpf_compute_subprog_arg_access(struct bpf_verifier_env *env);
 
 int bpf_stack_liveness_init(struct bpf_verifier_env *env);
diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index 52c39bbee6b1..a359e457e6c0 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -1715,6 +1715,8 @@ static int record_arg_access(struct bpf_verifier_env *env,
 		info = bpf_helper_stack_access_bytes(env, insn, arg_idx, insn_idx);
 	} else if (bpf_pseudo_kfunc_call(insn)) {
 		info = bpf_kfunc_stack_access_bytes(env, insn, arg_idx, insn_idx);
+	} else if (bpf_pseudo_call(insn)) {
+		info = bpf_global_subprog_stack_access_bytes(env, insn, arg_idx, insn_idx);
 	} else {
 		for (int f = 0; f <= depth; f++) {
 			err = mark_stack_read_all(env, instance, f, insn_idx);
@@ -1736,7 +1738,7 @@ static int record_arg_access(struct bpf_verifier_env *env,
 	return err;
 }
 
-/* Record stack access for a given 'at' state of helper/kfunc 'insn' */
+/* Record stack access for a given 'at' state of helper/kfunc/global subprog 'insn' */
 static int record_call_access(struct bpf_verifier_env *env,
 			      struct func_instance *instance,
 			      struct arg_track *at,
@@ -1744,12 +1746,19 @@ static int record_call_access(struct bpf_verifier_env *env,
 {
 	struct bpf_insn *insn = &env->prog->insnsi[insn_idx];
 	struct bpf_call_summary cs;
-	int r, err, arg_slot_cnt = 5;
+	int r, err, callee, arg_slot_cnt = 5;
 
-	if (bpf_pseudo_call(insn))
-		return 0;
-
-	if (bpf_is_callx(insn))
+	if (bpf_pseudo_call(insn)) {
+		/*
+		 * analyze_subprog() handles static callees. For global callees,
+		 * derive effects from argument types so may_write marks remain
+		 * conservative under freplace.
+		 */
+		callee = bpf_find_subprog(env, insn_idx + insn->imm + 1);
+		if (callee < 0 || !bpf_subprog_is_global(env, callee))
+			return 0;
+		arg_slot_cnt = MAX_BPF_FUNC_ARGS;
+	} else if (bpf_is_callx(insn))
 		/*
 		 * The callee is not known statically. Assume that all arg
 		 * slots are passed and let record_arg_access() conservatively
@@ -2282,7 +2291,12 @@ static int analyze_subprog(struct bpf_verifier_env *env,
 		if (bpf_pseudo_call(insn)) {
 			target = idx + insn->imm + 1;
 			callee = bpf_find_subprog(env, target);
-			if (callee < 0)
+			/*
+			 * Global subprograms can be freplaced, so conservatively derive
+			 * their stack effects from their BTF signatures instead of
+			 * analyzing them in the context of their callers.
+			 */
+			if (callee < 0 || bpf_subprog_is_global(env, callee))
 				continue;
 
 			/* Build entry args: R1-R5 and stack args from at_in at call site */
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 8987c9a69d10..a5456a6e2694 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -14571,6 +14571,72 @@ bpf_kfunc_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn *insn
 	return info;
 }
 
+/*
+ * Called before dead code elimination, some subprograms may be genuinely unreachable.
+ * Suppress diagnostics so callers can use a conservative result on error.
+ * The main pass would report errors for reachable subprograms.
+ */
+static bool try_btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
+{
+	struct bpf_prog *prog = env->prog;
+	u32 log_level = env->log.level;
+	int err;
+
+	if (!prog->aux->func_info || !prog->aux->func_info[subprog].type_id)
+		return false;
+
+	env->log.level = 0;
+	err = btf_prepare_func_args(env, subprog);
+	env->log.level = log_level;
+	return !err;
+}
+
+/*
+ * Derive stack effects for a global subprog's argument slot @arg (0-based)
+ * from its declared type, matching the main pass's argument checks.
+ */
+struct arg_access_info
+bpf_global_subprog_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn *insn,
+				      int arg, int insn_idx)
+{
+	struct arg_access_info info = {
+		.size = U32_MAX,
+		.may_read = true,
+		.may_write = true,
+	};
+	struct bpf_subprog_info *sub;
+	int callee;
+
+	callee = bpf_find_subprog(env, insn_idx + insn->imm + 1);
+	if (callee < 0)
+		return info;
+	sub = subprog_info(env, callee);
+	if (!try_btf_prepare_func_args(env, callee))
+		return info;
+
+	switch (base_type(sub->args[arg].arg_type)) {
+	case ARG_UNUSED:	/* Not an argument slot. */
+	case ARG_ANYTHING:	/* A scalar, the callee can't dereference it. */
+		return (struct arg_access_info) {};
+	case ARG_PTR_TO_BTF_ID:
+		/*
+		 * Trusted arguments cannot be stack pointers.
+		 * Untrusted arguments use ARG_IGNORE:
+		 * probe reads need no caller stack state, writes are prohibited.
+		 */
+		return (struct arg_access_info) {};
+	case ARG_PTR_TO_MEM:
+		/* mem_size is zero for untrusted pointers of unknown size. */
+		if (sub->args[arg].mem_size) {
+			info.size = sub->args[arg].mem_size;
+			info.must_write = true;
+		}
+		return info;
+	default:
+		return info;
+	}
+}
+
 /* check special kfuncs and return:
  *  1  - not fall-through to 'else' branch, continue verification
  *  0  - fall-through to 'else' branch

-- 
2.53.0

Re: [PATCH bpf-next v3 10/51] bpf: derive global subprog stack liveness from BTF signatures

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 3303 bytes --]

> bpf: derive global subprog stack liveness from BTF signatures

TL;DR: bpf_global_subprog_stack_access_bytes() likely matches ARG_ANYTHING
where btf_prepare_func_args() assigns ARG_SCALAR, so scalar args may get the
conservative read/write-all result; safe but less precise liveness.

> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index 8987c9a69d10..a5456a6e2694 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c

[ ... ]

> @@ -14571,6 +14571,72 @@ bpf_kfunc_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn *insn
>  	return info;
>  }
>

[ ... ]

> +struct arg_access_info
> +bpf_global_subprog_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn *insn,
> +				      int arg, int insn_idx)
> +{

[ ... ]

> +	switch (base_type(sub->args[arg].arg_type)) {
> +	case ARG_UNUSED:	/* Not an argument slot. */
> +	case ARG_ANYTHING:	/* A scalar, the callee can't dereference it. */
> +		return (struct arg_access_info) {};
                 ^^^^

Can this case ARG_ANYTHING ever match?  btf_prepare_func_args() doesn't
assign ARG_ANYTHING to a subprog argument.  It encodes scalar parameters,
including both halves of a 16-byte struct passed by value, as ARG_SCALAR:

kernel/bpf/btf.c:btf_prepare_func_args() {
    ...
            while (nslots--)
                    sub->args[slots_used++].arg_type = ARG_SCALAR;
    ...
}

do_check_common() and the exception callback check test for ARG_SCALAR too,
and ARG_SCALAR and ARG_ANYTHING are separate values in enum bpf_arg_type.

A scalar argument slot would therefore fall through to the default label,
which returns the conservative result (size = U32_MAX, may_read, may_write)
rather than the empty access described by the comment.

For example, with allow_ptr_leaks the main pass accepts a stack-derived
scalar here:

    r1 = r10
    r2 = r10
    r2 += -8
    r1 -= r2
    call global_func(int x)

Pointer minus pointer becomes an unknown scalar, so the ARG_SCALAR check
passes.  arg_track_alu64() keeps the frame identity (dst->frame ==
src->frame sets dst->off_cnt = 0), so arg_is_fp() is still true for r1.

record_arg_access() then calls record_stack_access() with off_cnt == 0,
which does mark_stack_read_all() and mark_stack_may_write_all() for the
whole caller frame at the call.  Before this patch the callee was analyzed
in context and made no stack accesses through its scalar parameter.

The result is safe but less precise.  More stack slots stay live, every
stack slot may be written at the call, and SCEV can no longer keep
stack-based recurrences across the call.

Should this be case ARG_SCALAR: instead?  That looks sound, since neither
the callee nor an freplace body can dereference a scalar argument.  No later
commit in the series (up to 726d0559f576) changes this.

> +	case ARG_PTR_TO_BTF_ID:
> +		/*
> +		 * Trusted arguments cannot be stack pointers.
> +		 * Untrusted arguments use ARG_IGNORE:
> +		 * probe reads need no caller stack state, writes are prohibited.
> +		 */
> +		return (struct arg_access_info) {};

[ ... ]


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/37923187511

Re: [PATCH bpf-next v3 10/51] bpf: derive global subprog stack liveness from BTF signatures

[Eduard Zingerman]

On Fri, 2026-10-09 at 12:07 +0000, bot+bpf-ci@kernel.org wrote:
> > bpf: derive global subprog stack liveness from BTF signatures
> 
> TL;DR: bpf_global_subprog_stack_access_bytes() likely matches ARG_ANYTHING
> where btf_prepare_func_args() assigns ARG_SCALAR, so scalar args may get the
> conservative read/write-all result; safe but less precise liveness.
> 
> > diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> > index 8987c9a69d10..a5456a6e2694 100644
> > --- a/kernel/bpf/verifier.c
> > +++ b/kernel/bpf/verifier.c
> 
> [ ... ]
> 
> > @@ -14571,6 +14571,72 @@ bpf_kfunc_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn *insn
> >  	return info;
> >  }
> > 
> 
> [ ... ]
> 
> > +struct arg_access_info
> > +bpf_global_subprog_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn *insn,
> > +				      int arg, int insn_idx)
> > +{
> 
> [ ... ]
> 
> > +	switch (base_type(sub->args[arg].arg_type)) {
> > +	case ARG_UNUSED:	/* Not an argument slot. */
> > +	case ARG_ANYTHING:	/* A scalar, the callee can't dereference it. */
> > +		return (struct arg_access_info) {};
>                  ^^^^
> 
> Can this case ARG_ANYTHING ever match?  btf_prepare_func_args() doesn't
> assign ARG_ANYTHING to a subprog argument.  It encodes scalar parameters,
> including both halves of a 16-byte struct passed by value, as ARG_SCALAR:
> 
> kernel/bpf/btf.c:btf_prepare_func_args() {
>     ...
>             while (nslots--)
>                     sub->args[slots_used++].arg_type = ARG_SCALAR;
>     ...
> }
> 
> do_check_common() and the exception callback check test for ARG_SCALAR too,
> and ARG_SCALAR and ARG_ANYTHING are separate values in enum bpf_arg_type.
> 
> A scalar argument slot would therefore fall through to the default label,
> which returns the conservative result (size = U32_MAX, may_read, may_write)
> rather than the empty access described by the comment.
> 
> For example, with allow_ptr_leaks the main pass accepts a stack-derived
> scalar here:
> 
>     r1 = r10
>     r2 = r10
>     r2 += -8
>     r1 -= r2
>     call global_func(int x)
> 
> Pointer minus pointer becomes an unknown scalar, so the ARG_SCALAR check
> passes.  arg_track_alu64() keeps the frame identity (dst->frame ==
> src->frame sets dst->off_cnt = 0), so arg_is_fp() is still true for r1.
> 
> record_arg_access() then calls record_stack_access() with off_cnt == 0,
> which does mark_stack_read_all() and mark_stack_may_write_all() for the
> whole caller frame at the call.  Before this patch the callee was analyzed
> in context and made no stack accesses through its scalar parameter.
> 
> The result is safe but less precise.  More stack slots stay live, every
> stack slot may be written at the call, and SCEV can no longer keep
> stack-based recurrences across the call.
> 
> Should this be case ARG_SCALAR: instead?  That looks sound, since neither
> the callee nor an freplace body can dereference a scalar argument.  No later
> commit in the series (up to 726d0559f576) changes this.

Valid point.
Should use ARG_SCALAR, can also add ARG_PTR_TO_ARENA.

...

[PATCH bpf-next v3 11/51] bpf: compute parameters liveness in bpf_compute_live_registers()

[Eduard Zingerman]

This computes exact liveness for the arguments of a call to a
subprogram instead of assuming that r1..r5 are read:
- For static subprograms compute use registers at the
  bpf_compute_live_registers() fixpoint phase.
- For global subprograms use BTF information.

Add a separate pass doing try_btf_prepare_func_args() in order for
bpf_subprog_info->arg_slot_cnt to be available during
bpf_compute_live_registers().

The conservative liveness marks hindered SCEV widening logic on a set
of production programs.

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 kernel/bpf/liveness.c                              | 31 ++++++++++++++++++++--
 kernel/bpf/verifier.c                              | 24 +++++++++++++++++
 .../testing/selftests/bpf/prog_tests/exceptions.c  |  2 +-
 .../selftests/bpf/progs/compute_live_registers.c   | 12 ++++-----
 .../selftests/bpf/progs/verifier_live_stack.c      |  2 +-
 5 files changed, 61 insertions(+), 10 deletions(-)

diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index a359e457e6c0..8d8428ccc505 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -2472,6 +2472,12 @@ static inline u32 mask_widen(u32 m) { return m | (m << 16); }
 static inline u16 mask_lo(u32 m) { return (u16)m; }
 static inline u16 mask_hi(u32 m) { return (u16)(m >> 16); }
 
+/* Index of the subprogram called by pseudo call @insn */
+static int callee_subprog(struct bpf_verifier_env *env, const struct bpf_insn *insn)
+{
+	return bpf_find_subprog(env, insn - env->prog->insnsi + insn->imm + 1);
+}
+
 /* Compute info->{use,def} fields for the instruction */
 static void compute_insn_live_regs(struct bpf_verifier_env *env,
 				   struct bpf_insn *insn,
@@ -2489,6 +2495,7 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
 	const u32 dst32 = mask_lo(dst);
 	const u32 r0  = reg64_mask(0);
 	const u32 r2  = reg64_mask(BPF_REG_2);
+	int subprog, reg_args;
 	u32 def = 0;
 	u32 use = U32_MAX;
 
@@ -2613,8 +2620,18 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
 		case BPF_CALL:
 			def = ALL_CALLER_SAVED_REGS;
 			use = def & ~BIT(BPF_REG_0);
-			if (bpf_get_call_summary(env, insn, &cs))
+			if (bpf_get_call_summary(env, insn, &cs)) {
 				use = GENMASK(min_t(u8, cs.arg_slot_cnt, MAX_BPF_FUNC_REG_ARGS), 1);
+			} else if (bpf_pseudo_call(insn)) {
+				/* a static callee's reads are added in bpf_compute_live_registers() */
+				subprog = callee_subprog(env, insn);
+				if (bpf_subprog_is_global(env, subprog)) {
+					reg_args = min_t(u8, env->subprog_info[subprog].arg_slot_cnt, MAX_BPF_FUNC_REG_ARGS);
+					use = GENMASK(reg_args, 1);
+				} else {
+					use = 0;
+				}
+			}
 			def = mask_widen(def);
 			use = mask_widen(use);
 			/* callx reads the address of the callee from dst_reg */
@@ -2699,11 +2716,21 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
 			struct bpf_iarray *succ;
 			u32 new_out = 0;
 			u32 new_in = 0;
+			u32 use;
 
 			succ = bpf_insn_successors(env, insn_idx);
 			for (int s = 0; s < succ->cnt; ++s)
 				new_out |= state[succ->items[s]].in;
-			new_in = (new_out & ~live->def) | live->use;
+			use = live->use;
+			/*
+			 * For a call to a static subprogram include the callee's
+			 * entry 'in' as computed so far.
+			 */
+			if (bpf_pseudo_call(&insns[insn_idx]) &&
+			    !bpf_subprog_is_global(env, callee_subprog(env, &insns[insn_idx])))
+				use |= state[insn_idx + insns[insn_idx].imm + 1].in &
+				       mask_widen(GENMASK(BPF_REG_5, BPF_REG_1));
+			new_in = (new_out & ~live->def) | use;
 			if (new_out != live->out || new_in != live->in) {
 				live->in = new_in;
 				live->out = new_out;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index a5456a6e2694..50cb564be9f7 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -10913,6 +10913,7 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, s
 	struct bpf_call_arg_meta meta;
 	struct bpf_func_proto *fn;
 	int ret, err;
+	u32 i;
 
 	memset(&meta, 0, sizeof(meta));
 	meta.btf = btf;
@@ -10939,6 +10940,15 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, s
 	meta.fn = fn;
 	meta.func_proto = func_proto;
 
+	/*
+	 * Check if all register arguments declared in BTF are initialized.
+	 * If they aren't the BTF is unreliable.
+	 */
+	if (!bpf_subprog_is_global(env, subprog)) {
+		for (i = 0; i < min_t(u32, sub->arg_slot_cnt, MAX_BPF_FUNC_REG_ARGS); i++)
+			if (caller->regs[BPF_REG_1 + i].type == NOT_INIT)
+				return -EINVAL;
+	}
 	return check_func_args(env, &meta, env->insn_idx);
 }
 
@@ -14591,6 +14601,18 @@ static bool try_btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
 	return !err;
 }
 
+/* Prepare argument metadata for liveness; the main program is handled separately. */
+static void prepare_subprog_args(struct bpf_verifier_env *env)
+{
+	int i;
+
+	for (i = 1; i < env->subprog_cnt; i++) {
+		if (!try_btf_prepare_func_args(env, i) && bpf_subprog_is_global(env, i))
+			/* Keep all argument registers live if the prototype is unavailable. */
+			env->subprog_info[i].arg_slot_cnt = MAX_BPF_FUNC_REG_ARGS;
+	}
+}
+
 /*
  * Derive stack effects for a global subprog's argument slot @arg (0-based)
  * from its declared type, matching the main pass's argument checks.
@@ -22836,6 +22858,8 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (ret < 0)
 		goto skip_full_check;
 
+	prepare_subprog_args(env);
+
 	ret = bpf_compute_live_registers(env);
 	if (ret < 0)
 		goto skip_full_check;
diff --git a/tools/testing/selftests/bpf/prog_tests/exceptions.c b/tools/testing/selftests/bpf/prog_tests/exceptions.c
index 3705db18c1bb..cea01d638f6f 100644
--- a/tools/testing/selftests/bpf/prog_tests/exceptions.c
+++ b/tools/testing/selftests/bpf/prog_tests/exceptions.c
@@ -322,7 +322,7 @@ static void test_exceptions_success(void)
 				       bpf_program__fd(skel->progs.exception_throw_subprog),
 				       "subprog"), "set_attach_target"))
 				goto done;
-		}), "subprog() is not a global function", 0);
+		}), "Cannot replace static functions", 0);
 
 	if (test__start_subtest("non-throwing extension -> throwing subprog"))
 		/* non-throwing extension -> throwing subprog : BAD (!global) */
diff --git a/tools/testing/selftests/bpf/progs/compute_live_registers.c b/tools/testing/selftests/bpf/progs/compute_live_registers.c
index 941b65fe7c07..b274a20ed7bd 100644
--- a/tools/testing/selftests/bpf/progs/compute_live_registers.c
+++ b/tools/testing/selftests/bpf/progs/compute_live_registers.c
@@ -416,11 +416,11 @@ static __used __naked int aux1(void)
 
 SEC("socket")
 __log_level(2)
-__msg("0: ....45.... (b7) r1 = 1")
-__msg("1: .1..45.... (b7) r2 = 2")
-__msg("2: .12.45.... (b7) r3 = 3")
-/* Conservative liveness for subprog parameters. */
-__msg("3: .12345.... (85) call pc+2")
+__msg("0: .......... (b7) r1 = 1")
+__msg("1: .1........ (b7) r2 = 2")
+__msg("2: .12....... (b7) r3 = 3")
+/* A static callee reads what its code reads: aux1 reads r1 and r2. */
+__msg("3: .12....... (85) call pc+2")
 __msg("4: .......... (b7) r0 = 0")
 __msg("5: 0......... (95) exit")
 __msg("6: .12....... (bf) r0 = r1")
@@ -452,7 +452,7 @@ SEC("socket")
 /* A program observing the pair needs the JIT; see bpf_compute_subprog_ret_regs(). */
 __load_if_JITed()
 __log_level(2)
-__msg("0: .12345.... (85) call pc+2")
+__msg("0: .......... (85) call pc+2")
 __msg("1: ..2....... (bf) r0 = r2")
 __msg("2: 0......... (95) exit")
 __msg("3: .......... (b7) r0 = 1")
diff --git a/tools/testing/selftests/bpf/progs/verifier_live_stack.c b/tools/testing/selftests/bpf/progs/verifier_live_stack.c
index 3ba21430bb75..3abe08cc6595 100644
--- a/tools/testing/selftests/bpf/progs/verifier_live_stack.c
+++ b/tools/testing/selftests/bpf/progs/verifier_live_stack.c
@@ -1470,7 +1470,7 @@ static __used __naked void inner_reader(void)
 SEC("socket")
 __log_level(2)
 /* fp-8 NOT live at call: callee writes before reading (parent_def kills it) */
-__msg("2: .12345.... (85) call pc+")
+__msg("2: .1........ (85) call pc+")
 __naked void callee_must_write_before_read(void)
 {
 	asm volatile (

-- 
2.53.0

Re: [PATCH bpf-next v3 11/51] bpf: compute parameters liveness in bpf_compute_live_registers()

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] bpf: state pruning bypasses NOT_INIT check for static subprog args
--

commit 9fa2cdb3328d1fdeb0246efe76e84a8ecaca85e6
Author: Eduard Zingerman <eddyz87@gmail.com>

bpf: compute parameters liveness in bpf_compute_live_registers()

This commit changes the verifier to compute exact liveness for the
arguments of calls to subprograms instead of assuming r1..r5 are read.
It adds a separate pass to prepare BTF function arguments early so that
liveness can be accurately determined.

> diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
> index a359e457e6c0c..8d8428ccc5052 100644
> --- a/kernel/bpf/liveness.c
> +++ b/kernel/bpf/liveness.c

[ ... ]

> @@ -2613,8 +2620,18 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
>  		case BPF_CALL:
>  			def = ALL_CALLER_SAVED_REGS;
>  			use = def & ~BIT(BPF_REG_0);
> -			if (bpf_get_call_summary(env, insn, &cs))
> +			if (bpf_get_call_summary(env, insn, &cs)) {
>  				use = GENMASK(min_t(u8, cs.arg_slot_cnt, MAX_BPF_FUNC_REG_ARGS), 1);
> +			} else if (bpf_pseudo_call(insn)) {
> +				/* a static callee's reads are added in bpf_compute_live_registers() */
> +				subprog = callee_subprog(env, insn);
> +				if (bpf_subprog_is_global(env, subprog)) {
> +					reg_args = min_t(u8, env->subprog_info[subprog].arg_slot_cnt, MAX_BPF_FUNC_REG_ARGS);
> +					use = GENMASK(reg_args, 1);
> +				} else {
> +					use = 0;
> +				}
> +			}

[Severity: High]
Does marking these unused arguments as not live for static subprograms
interact poorly with verifier state pruning? 

> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index a5456a6e26946..50cb564be9f74 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c

[ ... ]

> @@ -10939,6 +10940,15 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, s
>  	meta.fn = fn;
>  	meta.func_proto = func_proto;
>  
> +	/*
> +	 * Check if all register arguments declared in BTF are initialized.
> +	 * If they aren't the BTF is unreliable.
> +	 */
> +	if (!bpf_subprog_is_global(env, subprog)) {
> +		for (i = 0; i < min_t(u32, sub->arg_slot_cnt, MAX_BPF_FUNC_REG_ARGS); i++)
> +			if (caller->regs[BPF_REG_1 + i].type == NOT_INIT)
> +				return -EINVAL;
> +	}
>  	return check_func_args(env, &meta, env->insn_idx);
>  }

[Severity: High]
Because unused arguments are no longer marked as live due to the change
in compute_insn_live_regs(), can this NOT_INIT check be completely
bypassed by regsafe()?

If the verifier first explores a path where the argument is initialized,
the subprogram is not marked unreliable. If it subsequently explores
another path where the argument is left NOT_INIT, this second path
will be pruned before evaluating the BPF_CALL instruction and hitting
btf_check_func_arg_match().

Could this result in the subprogram remaining "reliable", allowing
fentry/freplace extension programs to attach to it and potentially
leak uninitialized kernel memory via the BTF context?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261009-scev-minimal-rebase-v3-0-bcba9bc53ac9@gmail.com?part=11

Re: [PATCH bpf-next v3 11/51] bpf: compute parameters liveness in bpf_compute_live_registers()

[Eduard Zingerman]

On Fri, 2026-10-09 at 11:26 +0000, sashiko-bot@kernel.org wrote:

...

> > @@ -10939,6 +10940,15 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, s
> >  	meta.fn = fn;
> >  	meta.func_proto = func_proto;
> >  
> > +	/*
> > +	 * Check if all register arguments declared in BTF are initialized.
> > +	 * If they aren't the BTF is unreliable.
> > +	 */
> > +	if (!bpf_subprog_is_global(env, subprog)) {
> > +		for (i = 0; i < min_t(u32, sub->arg_slot_cnt, MAX_BPF_FUNC_REG_ARGS); i++)
> > +			if (caller->regs[BPF_REG_1 + i].type == NOT_INIT)
> > +				return -EINVAL;
> > +	}
> >  	return check_func_args(env, &meta, env->insn_idx);
> >  }
> 
> [Severity: High]
> Because unused arguments are no longer marked as live due to the change
> in compute_insn_live_regs(), can this NOT_INIT check be completely
> bypassed by regsafe()?
> 
> If the verifier first explores a path where the argument is initialized,
> the subprogram is not marked unreliable. If it subsequently explores
> another path where the argument is left NOT_INIT, this second path
> will be pruned before evaluating the BPF_CALL instruction and hitting
> btf_check_func_arg_match().
> 
> Could this result in the subprogram remaining "reliable", allowing
> fentry/freplace extension programs to attach to it and potentially
> leak uninitialized kernel memory via the BTF context?

This is a valid point.
I should OR live[subprog_entry].in with BTF type mask.

[PATCH bpf-next v3 12/51] bpf: add tnum_alignment()

[Eduard Zingerman]

Used by the SCEV widening logic later in the series.
The verifier checks alignment for some memory accesses.
When SCEV widens a pointer or an index variable, it must retain
alignment information in order to allow array access.

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 include/linux/tnum.h |  2 ++
 kernel/bpf/tnum.c    | 11 +++++++++++
 2 files changed, 13 insertions(+)

diff --git a/include/linux/tnum.h b/include/linux/tnum.h
index ca2cfec8de08..16682248f963 100644
--- a/include/linux/tnum.h
+++ b/include/linux/tnum.h
@@ -134,4 +134,6 @@ static inline bool tnum_subreg_is_const(struct tnum a)
 /* Returns the smallest member of t larger than z */
 u64 tnum_step(struct tnum t, u64 z);
 
+u32 tnum_alignment(struct tnum a);
+
 #endif /* _LINUX_TNUM_H */
diff --git a/kernel/bpf/tnum.c b/kernel/bpf/tnum.c
index ec9c310cf5d7..1f620facd1b7 100644
--- a/kernel/bpf/tnum.c
+++ b/kernel/bpf/tnum.c
@@ -317,3 +317,14 @@ u64 tnum_step(struct tnum t, u64 z)
 	inc = (filled + 1) & t.mask;
 	return t.value | inc;
 }
+
+/*
+ * Return the number of trailing bits known to be zero in a tnum.
+ * Return 64 for a tnum representing zero.
+ */
+u32 tnum_alignment(struct tnum a)
+{
+	u64 v = a.value | a.mask;
+
+	return v ? __ffs64(v) : 64;
+}

-- 
2.53.0

[PATCH bpf-next v3 13/51] bpf: add cnum{32,64}_union()

[Eduard Zingerman]

Add cnum32_union() and cnum64_union() to compute a smallest circular
range containing both inputs.

Do so by enumerating the following configurations in a rotated
frame (a.base at the origin):

0                                              UT_MAX
|---------------------------------------------------|
[= a ==============================]                |
[= b tail =]           [= b main ===================>
[= union ===========================================]

0                                              UT_MAX
|---------------------------------------------------|
[= a =====================]                         |
[= b tail ======]                    [= b main =====>
[= union tail ============]          [= union main =>

0                                              UT_MAX
|---------------------------------------------------|
[= a =======]                                       |
[= b tail =============]             [= b main =====>
[= union tail =========]             [= union main =>

0                                              UT_MAX
|---------------------------------------------------|
[= a =========================]                     |
|                 [= b ====================]        |
[= union ==================================]        |

0                                              UT_MAX
|---------------------------------------------------|
[= a =====================================]         |
|               [= b ========]                      |
[= union =================================]         |

0                                              UT_MAX
|---------------------------------------------------|
[= a ============]                                  |
|                               [= b =======]       |

Two possible covering arcs:

[= ab ======================================]       |
[= ba tail ======]              [= ba main =========>

Pick the smaller one.

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 include/linux/cnum.h   |  2 ++
 kernel/bpf/cnum_defs.h | 44 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 46 insertions(+)

diff --git a/include/linux/cnum.h b/include/linux/cnum.h
index 49b7d0c7645d..ddf4726841e6 100644
--- a/include/linux/cnum.h
+++ b/include/linux/cnum.h
@@ -39,6 +39,7 @@ u32 cnum32_umin(struct cnum32 cnum);
 u32 cnum32_umax(struct cnum32 cnum);
 s32 cnum32_smin(struct cnum32 cnum);
 s32 cnum32_smax(struct cnum32 cnum);
+struct cnum32 cnum32_union(struct cnum32 a, struct cnum32 b);
 struct cnum32 cnum32_intersect(struct cnum32 a, struct cnum32 b);
 void cnum32_intersect_with(struct cnum32 *dst, struct cnum32 src);
 void cnum32_intersect_with_urange(struct cnum32 *dst, u32 min, u32 max);
@@ -65,6 +66,7 @@ u64 cnum64_umin(struct cnum64 cnum);
 u64 cnum64_umax(struct cnum64 cnum);
 s64 cnum64_smin(struct cnum64 cnum);
 s64 cnum64_smax(struct cnum64 cnum);
+struct cnum64 cnum64_union(struct cnum64 a, struct cnum64 b);
 struct cnum64 cnum64_intersect(struct cnum64 a, struct cnum64 b);
 void cnum64_intersect_with(struct cnum64 *dst, struct cnum64 src);
 void cnum64_intersect_with_urange(struct cnum64 *dst, u64 min, u64 max);
diff --git a/kernel/bpf/cnum_defs.h b/kernel/bpf/cnum_defs.h
index 30685e43de04..05b17a8b4814 100644
--- a/kernel/bpf/cnum_defs.h
+++ b/kernel/bpf/cnum_defs.h
@@ -198,6 +198,50 @@ static inline struct cnum_t FN(normalize)(struct cnum_t cnum)
 	return cnum;
 }
 
+/*
+ * Return a smallest arc containing both 'a' and 'b'.
+ * Break equal-size ties by choosing the smaller base.
+ */
+struct cnum_t FN(union)(struct cnum_t a, struct cnum_t b)
+{
+	struct cnum_t b1, ab, ba;
+	ut end;
+
+	if (FN(is_empty)(a))
+		return b;
+	if (FN(is_empty)(b))
+		return a;
+
+	/*
+	 * Rotate so that a1.base == 0 and a1.end == a.size.
+	 * Normalize b1 to preserve the full-circle representation.
+	 */
+	b1 = FN(normalize)((struct cnum_t){ b.base - a.base, b.size });
+	end = max(a.size, (ut)(b1.base + b1.size));
+
+	if (FN(urange_overflow)(b1)) {
+		/* a1 reaches b1's main arc: together they cover the circle. */
+		if (b1.base <= a.size)
+			return (struct cnum_t){ 0, UT_MAX };
+
+		/* Extend b1's tail through a1's end, then rotate back. */
+		return FN(normalize)((struct cnum_t){ b.base, end - b1.base });
+	}
+
+	/* ab, rotated back, covers both nonwrapping arcs. */
+	ab = (struct cnum_t){ a.base, end };
+	if (b1.base <= a.size)
+		return FN(normalize)(ab);
+
+	/* The arcs are disjoint; ba is the other possible covering arc. */
+	ba = (struct cnum_t){ b.base, a.size - b1.base };
+	if (ba.size < ab.size ||
+	    (ba.size == ab.size && ba.base < ab.base))
+		ab = ba;
+
+	return FN(normalize)(ab);
+}
+
 struct cnum_t FN(add)(struct cnum_t a, struct cnum_t b)
 {
 	if (FN(is_empty)(a) || FN(is_empty)(b))

-- 
2.53.0

[PATCH bpf-next v3 14/51] bpf: add cnum64_intersect_linear()

[Eduard Zingerman]

Add cnum64_intersect_linear(), which intersects a cnum64 interval with the
set of integers congruent to 'base' modulo 'step', i.e. the points
base + step * k. It tightens the interval by rounding its signed minimum up
and its signed maximum down to the nearest such point, and returns an empty
cnum when no point falls within the interval.

The arithmetic is done on the signed bounds [smin, smax]. The line is defined
over the integers, while a cnum64 is an arc on the u64 circle, so a u64-modular
computation would misplace the line for negative values whenever 'step' does
not divide 2^64. Working from the signed bounds keeps the residues consistent,
e.g. it can reason about a range like [-9, 3] with step 3.

Also add imod() to bpf_verifier.h: a mathematical modulo returning an
always-non-negative residue (unlike C's '%', which follows the sign of the
dividend). It is used by cnum64_intersect_linear() and by the base/step
tracking added in subsequent patches.

This is a building block for tracking scalar registers whose value is known
to lie on a line base + step * k.

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 include/linux/bpf_verifier.h | 14 ++++++++++++++
 include/linux/cnum.h         |  1 +
 kernel/bpf/cnum.c            | 36 ++++++++++++++++++++++++++++++++++++
 3 files changed, 51 insertions(+)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 05e6f091ad94..0422d4a1be01 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -9,6 +9,20 @@
 #include <linux/filter.h> /* for MAX_BPF_STACK */
 #include <linux/tnum.h>
 #include <linux/cnum.h>
+#include <linux/math64.h> /* for div_s64_rem() */
+
+/*
+ * Mathematical modulo, the residue of 'v' modulo 'step', normalized to [0, step).
+ * This differs from C's '%' operator, which truncates the quotient toward zero
+ * and so returns a remainder with the sign of the dividend (e.g. -1 % 3 == -1, not 2).
+ */
+static inline u16 imod(s64 v, u16 step)
+{
+	s32 rem;
+
+	div_s64_rem(v, step, &rem);
+	return rem < 0 ? rem + step : rem;
+}
 
 /* Maximum variable offset umax_value permitted when resolving memory accesses.
  * In practice this is far bigger than any realistic pointer offset; this limit
diff --git a/include/linux/cnum.h b/include/linux/cnum.h
index ddf4726841e6..49160318adf9 100644
--- a/include/linux/cnum.h
+++ b/include/linux/cnum.h
@@ -80,5 +80,6 @@ bool cnum64_is_subset(struct cnum64 outer, struct cnum64 inner);
 
 struct cnum32 cnum32_from_cnum64(struct cnum64 cnum);
 struct cnum64 cnum64_cnum32_intersect(struct cnum64 a, struct cnum32 b);
+struct cnum64 cnum64_intersect_linear(struct cnum64 in, u16 base, u16 step);
 
 #endif /* _LINUX_CNUM_H */
diff --git a/kernel/bpf/cnum.c b/kernel/bpf/cnum.c
index 86142cb2aee5..8ec5187d7c75 100644
--- a/kernel/bpf/cnum.c
+++ b/kernel/bpf/cnum.c
@@ -2,6 +2,7 @@
 /* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
 
 #include <linux/bits.h>
+#include <linux/bpf_verifier.h> /* for imod() */
 
 #define T 32
 #include "cnum_defs.h"
@@ -118,3 +119,38 @@ struct cnum64 cnum64_cnum32_intersect(struct cnum64 a, struct cnum32 b)
 	}
 	return t;
 }
+
+/* Intersect 'in' with the set of integers defined by equation 'base + step * k'. */
+struct cnum64 cnum64_intersect_linear(struct cnum64 in, u16 base, u16 step)
+{
+	s64 smin = cnum64_smin(in);
+	s64 smax = cnum64_smax(in);
+	s64 lo, hi;
+	u16 d;
+
+	if (step <= 1 || cnum64_is_empty(in) || cnum64_srange_overflow(in))
+		return in;
+	/*
+	 * Round smin up to the next value congruent to 'base' modulo 'step',
+	 * i.e. increase smin by d = (base - smin) mod step:
+	 *
+	 *                 |<---- d ---->|
+	 *     |-----------|=============|...
+	 * base+step*k    smin       base+step*(k+1)
+	 */
+	d = imod(base - imod(smin, step), step);
+	if ((u64)smax - (u64)smin < d)
+		return CNUM64_EMPTY;
+	lo = smin + d;
+	/*
+	 * Round smax down to the previous value congruent to 'base' modulo 'step',
+	 * i.e. decrease smax by d = (smax - base) mod step:
+	 *
+	 *     |<--- d --->|
+	 *  ...|===========|-------------|
+	 * base+step*k    smax       base+step*(k+1)
+	 */
+	d = imod(imod(smax, step) - base, step);
+	hi = smax - d;
+	return cnum64_from_srange(lo, hi);
+}

-- 
2.53.0

[PATCH bpf-next v3 15/51] bpf: expose bpf_rev_opcode() and bpf_flip_opcode()

[Eduard Zingerman]

Expose the existing helpers verifier.c:rev_opcode() and
verifier.c:flip_opcode() as bpf_rev_opcode() and bpf_flip_opcode()
and declare them in bpf_verifier.h. The helpers are used by SCEV
logic while analyzing loop conditions.

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 include/linux/bpf_verifier.h |  3 +++
 kernel/bpf/verifier.c        | 16 +++++++---------
 2 files changed, 10 insertions(+), 9 deletions(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 0422d4a1be01..5ec9495d42b1 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1875,4 +1875,7 @@ int bpf_fixup_call_args(struct bpf_verifier_env *env);
 int bpf_do_misc_fixups(struct bpf_verifier_env *env);
 int bpf_insn_def32(struct bpf_prog *prog, struct bpf_insn *insn);
 
+int bpf_flip_opcode(u32 opcode);
+u8 bpf_rev_opcode(u8 opcode);
+
 #endif /* _LINUX_BPF_VERIFIER_H */
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 50cb564be9f7..e317cdae749e 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -17333,8 +17333,6 @@ static void find_good_pkt_pointers(struct bpf_verifier_state *vstate,
 
 static void regs_refine_cond_op(struct bpf_reg_state *reg1, struct bpf_reg_state *reg2,
 				u8 opcode, bool is_jmp32);
-static u8 rev_opcode(u8 opcode);
-
 /*
  * Learn more information about live branches by simulating refinement on both branches.
  * regs_refine_cond_op() is sound, so producing ill-formed register bounds for the branch means
@@ -17343,7 +17341,7 @@ static u8 rev_opcode(u8 opcode);
 static int simulate_both_branches_taken(struct bpf_verifier_env *env, u8 opcode, bool is_jmp32)
 {
 	/* Fallthrough (FALSE) branch */
-	regs_refine_cond_op(&env->false_reg1, &env->false_reg2, rev_opcode(opcode), is_jmp32);
+	regs_refine_cond_op(&env->false_reg1, &env->false_reg2, bpf_rev_opcode(opcode), is_jmp32);
 	reg_bounds_sync(&env->false_reg1);
 	reg_bounds_sync(&env->false_reg2);
 	/*
@@ -17529,7 +17527,7 @@ static int is_scalar_branch_taken(struct bpf_verifier_env *env, struct bpf_reg_s
 	return simulate_both_branches_taken(env, opcode, is_jmp32);
 }
 
-static int flip_opcode(u32 opcode)
+int bpf_flip_opcode(u32 opcode)
 {
 	/* How can we transform "a <op> b" into "b <op> a"? */
 	static const u8 opcode_flip[16] = {
@@ -17560,7 +17558,7 @@ static int is_pkt_ptr_branch_taken(struct bpf_reg_state *dst_reg,
 		pkt = dst_reg;
 	} else if (dst_reg->type == PTR_TO_PACKET_END) {
 		pkt = src_reg;
-		opcode = flip_opcode(opcode);
+		opcode = bpf_flip_opcode(opcode);
 	} else {
 		return -1;
 	}
@@ -17615,7 +17613,7 @@ static int is_branch_taken(struct bpf_verifier_env *env, struct bpf_reg_state *r
 
 		/* arrange that reg2 is a scalar, and reg1 is a pointer */
 		if (!is_reg_const(reg2, is_jmp32)) {
-			opcode = flip_opcode(opcode);
+			opcode = bpf_flip_opcode(opcode);
 			swap(reg1, reg2);
 		}
 		/* and ensure that reg2 is a constant */
@@ -17649,7 +17647,7 @@ static int is_branch_taken(struct bpf_verifier_env *env, struct bpf_reg_state *r
 /* Opcode that corresponds to a *false* branch condition.
  * E.g., if r1 < r2, then reverse (false) condition is r1 >= r2
  */
-static u8 rev_opcode(u8 opcode)
+u8 bpf_rev_opcode(u8 opcode)
 {
 	switch (opcode) {
 	case BPF_JEQ:		return BPF_JNE;
@@ -17684,7 +17682,7 @@ static void regs_refine_cond_op(struct bpf_reg_state *reg1, struct bpf_reg_state
 	case BPF_JGT:
 	case BPF_JSGE:
 	case BPF_JSGT:
-		opcode = flip_opcode(opcode);
+		opcode = bpf_flip_opcode(opcode);
 		swap(reg1, reg2);
 		break;
 	default:
@@ -17751,7 +17749,7 @@ static void regs_refine_cond_op(struct bpf_reg_state *reg1, struct bpf_reg_state
 			reg1->var_off = tnum_or(reg1->var_off, tnum_const(val));
 		}
 		break;
-	case BPF_JSET | BPF_X: /* reverse of BPF_JSET, see rev_opcode() */
+	case BPF_JSET | BPF_X: /* reverse of BPF_JSET, see bpf_rev_opcode() */
 		if (!is_reg_const(reg2, is_jmp32))
 			swap(reg1, reg2);
 		if (!is_reg_const(reg2, is_jmp32))

-- 
2.53.0

[PATCH bpf-next v3 16/51] bpf: expose bpf_grow_stack_state()

[Eduard Zingerman]

When windening SCEV expressions like *fp-8 -> (any *fp-8 *fp-16),
fp-8 might be not yet allocated. Expose a utility function such
that widening logic can handle this.

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 include/linux/bpf_verifier.h | 3 ++-
 kernel/bpf/verifier.c        | 4 ++--
 2 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 5ec9495d42b1..ab86b75888c3 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -522,7 +522,7 @@ static inline u32 bpf_stack_nr_slots(const struct bpf_func_state *frame)
 
 /*
  * Stack slot @spi of @frame, covering bytes [fp - (spi + 1) * 8, fp - spi * 8).
- * The caller must ensure spi < bpf_stack_nr_slots(frame), see grow_stack_state().
+ * The caller must ensure spi < bpf_stack_nr_slots(frame), see bpf_grow_stack_state().
  */
 static inline struct bpf_stack_state *bpf_stack_slot(const struct bpf_func_state *frame, u32 spi)
 {
@@ -1297,6 +1297,7 @@ void bpf_bt_sync_linked_regs(struct backtrack_state *bt, struct bpf_jmp_history_
 void bpf_mark_reg_not_init(const struct bpf_verifier_env *env,
 			   struct bpf_reg_state *reg);
 void bpf_mark_reg_unknown_imprecise(struct bpf_reg_state *reg);
+int bpf_grow_stack_state(struct bpf_verifier_env *env, struct bpf_func_state *state, int size);
 void bpf_mark_all_scalars_precise(struct bpf_verifier_env *env,
 				  struct bpf_verifier_state *st);
 void bpf_clear_singular_ids(struct bpf_verifier_env *env, struct bpf_verifier_state *st);
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index e317cdae749e..47a632a6749a 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -1456,7 +1456,7 @@ static int resize_reference_state(struct bpf_verifier_state *state, size_t n)
 /* Possibly update state->allocated_stack to be at least size bytes. Also
  * possibly update the function's high-water mark in its bpf_subprog_info.
  */
-static int grow_stack_state(struct bpf_verifier_env *env, struct bpf_func_state *state, int size)
+int bpf_grow_stack_state(struct bpf_verifier_env *env, struct bpf_func_state *state, int size)
 {
 	size_t old_n = bpf_stack_nr_slots(state), n;
 
@@ -6705,7 +6705,7 @@ static int check_stack_access_within_bounds(
 	/* Note that there is no stack access with offset zero, so the needed stack
 	 * size is -min_off, not -min_off+1.
 	 */
-	return grow_stack_state(env, state, -min_off /* size */);
+	return bpf_grow_stack_state(env, state, -min_off /* size */);
 }
 
 static bool get_func_retval_range(struct bpf_prog *prog,

-- 
2.53.0

[PATCH bpf-next v3 17/51] bpf: expose bpf_verbose_snum()

[Eduard Zingerman]

For use in SCEV logging to make large widened bounds more readable.

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 include/linux/bpf_verifier.h |  1 +
 kernel/bpf/log.c             | 12 ++++++------
 2 files changed, 7 insertions(+), 6 deletions(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index ab86b75888c3..b4396867c6f3 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1165,6 +1165,7 @@ int bpf_vlog_finalize(struct bpf_verifier_log *log, u32 *log_size_actual);
 __printf(3, 4) void verbose_linfo(struct bpf_verifier_env *env,
 				  u32 insn_off,
 				  const char *prefix_fmt, ...);
+void bpf_verbose_snum(struct bpf_verifier_env *env, s64 num);
 
 #define verifier_bug_if(cond, env, fmt, args...)						\
 	({											\
diff --git a/kernel/bpf/log.c b/kernel/bpf/log.c
index d850a7863d2e..87d137734db1 100644
--- a/kernel/bpf/log.c
+++ b/kernel/bpf/log.c
@@ -527,7 +527,7 @@ static void verbose_unum(struct bpf_verifier_env *env, u64 num)
 		verbose(env, "%#llx", num);
 }
 
-static void verbose_snum(struct bpf_verifier_env *env, s64 num)
+void bpf_verbose_snum(struct bpf_verifier_env *env, s64 num)
 {
 	if (is_snum_decimal(num))
 		verbose(env, "%lld", num);
@@ -609,7 +609,7 @@ static void print_scalar_ranges(struct bpf_verifier_env *env,
 		}
 
 		if (m1->name[0] == 's')
-			verbose_snum(env, m1->val);
+			bpf_verbose_snum(env, m1->val);
 		else
 			verbose_unum(env, m1->val);
 	}
@@ -632,7 +632,7 @@ static void print_reg_state(struct bpf_verifier_env *env,
 	if (t == SCALAR_VALUE && reg->precise)
 		verbose(env, "P");
 	if (t == SCALAR_VALUE && tnum_is_const(reg->var_off)) {
-		verbose_snum(env, reg->var_off.value);
+		bpf_verbose_snum(env, reg->var_off.value);
 		return;
 	}
 
@@ -643,7 +643,7 @@ static void print_reg_state(struct bpf_verifier_env *env,
 		if (state->frameno != reg->frameno)
 			verbose(env, "[%d]", reg->frameno);
 		if (tnum_is_const(reg->var_off)) {
-			verbose_snum(env, reg->var_off.value + reg->delta);
+			bpf_verbose_snum(env, reg->var_off.value + reg->delta);
 			return;
 		}
 	}
@@ -667,7 +667,7 @@ static void print_reg_state(struct bpf_verifier_env *env,
 	}
 	if (t != SCALAR_VALUE && reg->delta) {
 		verbose_a("off=");
-		verbose_snum(env, reg->delta);
+		bpf_verbose_snum(env, reg->delta);
 	}
 	if (type_is_pkt_pointer(t)) {
 		verbose_a("r=");
@@ -683,7 +683,7 @@ static void print_reg_state(struct bpf_verifier_env *env,
 		/* a pointer register with fixed offset */
 		if (reg->var_off.value) {
 			verbose_a("imm=");
-			verbose_snum(env, reg->var_off.value);
+			bpf_verbose_snum(env, reg->var_off.value);
 		}
 	} else {
 		print_scalar_ranges(env, reg, &sep);

-- 
2.53.0

[PATCH bpf-next v3 18/51] bpf: allow subrange relations for PTR_TO_STACK in regsafe()

[Eduard Zingerman]

regsafe() currently requires an exact match for PTR_TO_STACK registers.
This prevents an explored state from pruning a current state whose
variable-offset range is a subset of the explored range.

This change allows a checkpoint containing a widened stack pointer to
cover narrower offsets reached on subsequent loop iterations,
for example [-56, -8] within [-64, -8].

Merge the PTR_TO_STACK case with PTR_TO_MEM and others, as the set of
necessary checks is almost identical. The only difference is a map_uid
check, which is not needed for PTR_TO_STACK. All locations that
produce PTR_TO_STACK reset map_uid to zero.

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 kernel/bpf/states.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/kernel/bpf/states.c b/kernel/bpf/states.c
index 18bf7b660c2f..345079cf64d3 100644
--- a/kernel/bpf/states.c
+++ b/kernel/bpf/states.c
@@ -610,6 +610,7 @@ static bool regsafe(struct bpf_verifier_env *env, struct bpf_reg_state *rold,
 	case PTR_TO_MEM:
 	case PTR_TO_BUF:
 	case PTR_TO_TP_BUFFER:
+	case PTR_TO_STACK:
 		/* If the new min/max/var_off satisfy the old ones and
 		 * everything else matches, we are OK.
 		 */
@@ -643,8 +644,6 @@ static bool regsafe(struct bpf_verifier_env *env, struct bpf_reg_state *rold,
 		/* new val must satisfy old val knowledge */
 		return range_within(rold, rcur) &&
 		       tnum_in(rold->var_off, rcur->var_off);
-	case PTR_TO_STACK:
-		return regs_exact(rold, rcur, idmap);
 	case PTR_TO_ARENA:
 		return true;
 	case PTR_TO_INSN:

-- 
2.53.0

[PATCH bpf-next v3 19/51] bpf: representation for intervals with steps

[Eduard Zingerman]

Extend scalar register tracking with a linear "base + step * k" description:
each scalar carries two u16 fields, base and step (invariant: base < step),
recording that its value is some point on the line base + step * k.
A fresh scalar starts as base=0, step=1, i.e. any integer.

This lets the verifier reason about strided values, e.g. an index multiplied
by an element size, which neither the min/max bounds nor the tnum can capture
for a non-power-of-2 stride. Such reasoning is necessary to allow
array access with a variable offset through PTR_TO_BTF_ID, e.g.
for expressions like p->arr[i].field.

reg_bounds_sync() uses the description in two ways:
- __reg_bound_offset() clears the low bits of var_off implied by the number
  of trailing zeros shared by base and step.
- __update_bounds_from_step() intersects the 64-bit range with the line via
  cnum64_intersect_linear(), snapping smin/smax to actual points on it.
  This is the last step of the sync, as refining the range from var_off
  can move the bounds off the line.

range_within() is extended to check whether the cur register's equation
describes a subset of the points described by the old register's equation.

Scalar ALU operations update base/step as follows:
- ADD of a constant (including pointer + scalar): keep step, shift base;
- MUL by a constant: scale step and base by the constant;
- LSH by a constant: scale step and base by the corresponding power of two;
- sign extension: keep the register unchanged if all values fit the target
  type, reset to base=0, step=1 otherwise;
- All other operations reset to base=0, step=1.

Branch (eq/neq) checks do not take into account base/step yet.

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 include/linux/bpf_verifier.h |   7 ++
 kernel/bpf/log.c             |   2 +
 kernel/bpf/states.c          |  26 +++++-
 kernel/bpf/verifier.c        | 211 ++++++++++++++++++++++++++++++++++++++++++-
 4 files changed, 240 insertions(+), 6 deletions(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index b4396867c6f3..2560754ed56c 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -139,6 +139,13 @@ struct bpf_reg_state {
 	 */
 	struct cnum64 r64; /* 64-bit range as circular number */
 	struct cnum32 r32; /* 32-bit range as circular number */
+	/*
+	 * The value described by this register, interpreted as s64, lies on
+	 * a line described by a linear equation base + step * k.
+	 * Invariant: base < step.
+	 */
+	u16 base;
+	u16 step;
 	/* For PTR_TO_PACKET, used to find other pointers with the same variable
 	 * offset, so they can share range knowledge.
 	 * For PTR_TO_MAP_VALUE_OR_NULL this is used to share which map value we
diff --git a/kernel/bpf/log.c b/kernel/bpf/log.c
index 87d137734db1..8ee7d7b71147 100644
--- a/kernel/bpf/log.c
+++ b/kernel/bpf/log.c
@@ -693,6 +693,8 @@ static void print_reg_state(struct bpf_verifier_env *env,
 			tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off);
 			verbose_a("var_off=%s", tn_buf);
 		}
+		if (reg->base != 0 || reg->step != 1)
+			verbose_a("step=%d+%d", reg->base, reg->step);
 	}
 	verbose(env, ")");
 }
diff --git a/kernel/bpf/states.c b/kernel/bpf/states.c
index 345079cf64d3..981328f5eb60 100644
--- a/kernel/bpf/states.c
+++ b/kernel/bpf/states.c
@@ -302,8 +302,30 @@ int bpf_update_branch_counts(struct bpf_verifier_env *env, struct bpf_verifier_s
 static bool range_within(const struct bpf_reg_state *old,
 			 const struct bpf_reg_state *cur)
 {
-	return cnum64_is_subset(old->r64, cur->r64) &&
-	       cnum32_is_subset(old->r32, cur->r32);
+	if (!cnum64_is_subset(old->r64, cur->r64) ||
+	    !cnum32_is_subset(old->r32, cur->r32))
+		return false;
+
+	if (old->step <= 1)
+		return true;
+
+	if (cnum64_is_const(cur->r64))
+		return imod((s64)cnum64_smin(cur->r64), old->step) == old->base;
+
+	/*
+	 * Both `old` and `cur` define some sets of points.
+	 * Return true, if points defined by `cur` are a subset of points defined by `old`:
+	 * - bounds for `cur` should be within bounds for `old`;
+	 * - cur->step should be dividable by old->step;
+	 * - cur->base should start at integer number of old->step
+	 *   steps from old->base.
+	 *
+	 * E.g. the following ranges are compatible:
+	 * - old [0, 10] step 2
+	 * - new [2, 10] step 4
+	 */
+	return cur->step % old->step == 0 &&
+	       (cur->base - old->base) % old->step == 0;
 }
 
 /* If in the old state two registers had the same id, then they need to have
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 47a632a6749a..f5b2500335a7 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -5,6 +5,7 @@
  */
 #include <uapi/linux/btf.h>
 #include <linux/bpf-cgroup.h>
+#include <linux/count_zeros.h>
 #include <linux/kernel.h>
 #include <linux/types.h>
 #include <linux/slab.h>
@@ -1922,12 +1923,124 @@ static void bpf_diag_record_caller_saved(struct bpf_verifier_env *env,
 	}
 }
 
+static void reg_step_reset(struct bpf_reg_state *reg)
+{
+	reg->base = 0;
+	reg->step = 1;
+}
+
+static void scalar_step_add(struct bpf_reg_state *dst_reg,
+			    const struct bpf_reg_state *a,
+			    const struct bpf_reg_state *b)
+{
+	const struct bpf_reg_state *reg;
+	s64 amount, tmp;
+
+	if (tnum_is_const(b->var_off)) {
+		reg = a;
+		amount = (s64)b->var_off.value;
+	} else if (tnum_is_const(a->var_off)) {
+		reg = b;
+		amount = (s64)a->var_off.value;
+	} else {
+		reg_step_reset(dst_reg);
+		return;
+	}
+
+	/*
+	 * Let x be a possible signed value of the register,
+	 * d = amount, s = reg->step, and b = reg->base.
+	 * Then x mod s = b, with s > 0 and 0 <= b < s.
+	 * If the addition causes signed underflow, the resulting value
+	 * is x + d + 2**64.
+	 *
+	 * Taking the result modulo s and substituting x mod s = b gives:
+	 *   (x + d + 2**64) mod s = (b + d + (2**64 mod s)) mod s      (1)
+	 * Keeping reg->step = s and updating reg->base to (b + d) mod s
+	 * requires (1) to equal (b + d) mod s, which holds if and only
+	 * if 2**64 mod s = 0.
+	 * Signed overflow subtracts 2**64 and has the same requirement.
+	 *
+	 * Conservatively reset the stride if either wrap is possible.
+	 */
+	if (check_add_overflow(reg_smin(reg), amount, &tmp) ||
+	    check_add_overflow(reg_smax(reg), amount, &tmp)) {
+		reg_step_reset(dst_reg);
+		return;
+	}
+
+	dst_reg->base = (reg->base + imod(amount, reg->step)) % reg->step;
+	dst_reg->step = reg->step;
+}
+
+static void scalar_step_scale(struct bpf_reg_state *dst_reg, u64 amount)
+{
+	u16 step;
+	s64 tmp;
+
+	/*
+	 * Using the same notation as scalar_step_add(), with d > 0,
+	 * the scaled value satisfies (x * d) mod (s * d) = b * d.
+	 * If the multiplication wraps, its signed result can be represented
+	 * as x * d - n * 2**64 for some nonzero integer n.
+	 *
+	 * Taking the result modulo s * d gives:
+	 *   (x * d - n * 2**64) mod (s * d) =
+	 *   (b * d - (n * 2**64 mod (s * d))) mod (s * d)              (1)
+	 * Updating dst_reg->base to b * d and dst_reg->step to s * d
+	 * requires (1) to equal b * d, which holds if and only if
+	 * n * 2**64 mod (s * d) = 0.
+	 *
+	 * Conservatively reset the stride if a wrap is possible.
+	 */
+	if (amount == 0 || check_mul_overflow(dst_reg->step, amount, &step) ||
+	    check_mul_overflow(reg_smin(dst_reg), (s64)amount, &tmp) ||
+	    check_mul_overflow(reg_smax(dst_reg), (s64)amount, &tmp)) {
+		reg_step_reset(dst_reg);
+		return;
+	}
+	/*
+	 * The above established that `step = dst_reg->step * amount`
+	 * does not overflow, therefore the base < step invariant holds
+	 * after assignments below.
+	 */
+	dst_reg->base = dst_reg->base * amount;
+	dst_reg->step = step;
+}
+
+/*
+ * Let x be a possible signed value of the register,
+ * s = reg->step, and b = reg->base.
+ * Then x mod s = b, with s > 0 and 0 <= b < s.
+ * Truncating to N bits and then zero- or sign-extending x
+ * is equivalent to computing x + k * 2**N for some integer k.
+ * For example, sign-extending the low 8 bits of x = 255 can
+ * be expressed as 255 - 2**8 = -1.
+ *
+ * Taking the result modulo s and substituting x mod s = b gives:
+ *   (x + k * 2**N) mod s = (b + (k * 2**N mod s)) mod s
+ * Keeping reg->base = b and reg->step = s requires this to equal b,
+ * which holds if and only if k * 2**N mod s = 0.
+ * Continuing the example with s = 3 and b = 0, we have 255 mod 3 = 0,
+ * but (-1) mod 3 = 2, so the original base and step no longer hold.
+ *
+ * Conservatively reset the stride unless all possible values fit in
+ * the range where the conversion leaves them unchanged.
+ * Call before updating the register's bounds. Requires 0 < bits < 64.
+ */
+static void reg_step_check_unsigned(struct bpf_reg_state *reg, u32 bits)
+{
+	if (reg_umax(reg) >= (1ULL << bits))
+		reg_step_reset(reg);
+}
+
 /* This helper doesn't clear reg->id */
 static void ___mark_reg_known(struct bpf_reg_state *reg, u64 imm)
 {
 	reg->var_off = tnum_const(imm);
 	reg->r64 = cnum64_from_urange(imm, imm);
 	reg->r32 = cnum32_from_urange((u32)imm, (u32)imm);
+	reg_step_reset(reg);
 }
 
 /* Mark the unknown part of a register (variable offset or scalar value) as
@@ -2185,8 +2298,24 @@ static void __reg_bound_offset(struct bpf_reg_state *reg)
 	struct tnum var32_off = tnum_intersect(tnum_subreg(var64_off),
 					       tnum_range(reg_u32_min(reg),
 							  reg_u32_max(reg)));
+	u32 trailing_zero_bits;
+	u16 base = reg->base;
+	u16 step = reg->step;
 
 	reg->var_off = tnum_or(tnum_clear_subreg(var64_off), var32_off);
+
+	if (base == 0)
+		trailing_zero_bits = count_trailing_zeros(step);
+	else
+		trailing_zero_bits = min(count_trailing_zeros(base),
+					 count_trailing_zeros(step));
+	reg->var_off = tnum_and(reg->var_off, tnum_const(~0ULL << trailing_zero_bits));
+}
+
+static void __update_bounds_from_step(struct bpf_reg_state *reg)
+{
+	reg->r64 = cnum64_intersect_linear(reg->r64, reg->base, reg->step);
+	deduce_bounds_32_from_64(reg);
 }
 
 static bool range_bounds_violation(struct bpf_reg_state *reg);
@@ -2208,6 +2337,15 @@ static void reg_bounds_sync(struct bpf_reg_state *reg)
 	 * then new var_off is (0; 0x7f...fc) which improves our umax.
 	 */
 	__update_reg_bounds(reg);
+	/*
+	 * Both tnum and base+step define a sequence of points,
+	 * potentially w/o intersection, or intersecting after N
+	 * tnum_step()'s, where N can be large. On the other hand,
+	 * step+base *must* be in sync with r64 in order for
+	 * array traversal logic starting at reg_umin() to be sound.
+	 * Hence, place this as a final step in the reg_bounds_sync().
+	 */
+	__update_bounds_from_step(reg);
 }
 
 static bool const_tnum_range_mismatch(struct bpf_reg_state *reg)
@@ -2251,17 +2389,24 @@ static int reg_bounds_sanity_check(struct bpf_verifier_env *env,
 		goto out;
 	}
 
+	if (reg->step == 0 || reg->base >= reg->step) {
+		msg = "invalid base/step";
+		goto out;
+	}
+
 	return 0;
 out:
 	verifier_bug(env, "REG INVARIANTS VIOLATION (%s): %s r64={.base=%#llx, .size=%#llx} "
-		     "r32={.base=%#x, .size=%#x} var_off=(%#llx, %#llx)",
+		     "r32={.base=%#x, .size=%#x} var_off=(%#llx, %#llx) step=%u+%u",
 		     ctx, msg,
 		     reg->r64.base, reg->r64.size,
 		     reg->r32.base, reg->r32.size,
-		     reg->var_off.value, reg->var_off.mask);
+		     reg->var_off.value, reg->var_off.mask,
+		     reg->base, reg->step);
 	if (env->test_reg_invariants)
 		return -EFAULT;
 	__mark_reg_unbounded(reg);
+	reg_step_reset(reg);
 	return 0;
 }
 
@@ -2272,6 +2417,7 @@ void bpf_mark_reg_unknown_imprecise(struct bpf_reg_state *reg)
 	reg->type = SCALAR_VALUE;
 	reg->var_off = tnum_unknown;
 	__mark_reg_unbounded(reg);
+	reg_step_reset(reg);
 }
 
 /* Mark a register as having a completely unknown (scalar) value,
@@ -5902,6 +6048,7 @@ static int check_buffer_access(struct bpf_verifier_env *env,
 /* BPF architecture zero extends alu32 ops into 64-bit registesr */
 static void zext_32_to_64(struct bpf_reg_state *reg)
 {
+	reg_step_check_unsigned(reg, 32);
 	reg->var_off = tnum_subreg(reg->var_off);
 	reg_set_urange64(reg, reg_u32_min(reg), reg_u32_max(reg));
 }
@@ -5911,13 +6058,14 @@ static void zext_32_to_64(struct bpf_reg_state *reg)
  */
 static void coerce_reg_to_size(struct bpf_reg_state *reg, int size)
 {
-	u64 mask;
+	u64 mask = (1ULL << (size * 8)) - 1;
+
+	reg_step_check_unsigned(reg, size * 8);
 
 	/* clear high bits in bit representation */
 	reg->var_off = tnum_cast(reg->var_off, size);
 
 	/* fix arithmetic bounds */
-	mask = ((u64)1 << (size * 8)) - 1;
 	if ((reg_umin(reg) & ~mask) == (reg_umax(reg) & ~mask))
 		reg_set_urange64(reg, reg_umin(reg) & mask, reg_umax(reg) & mask);
 	else
@@ -5954,6 +6102,14 @@ static void coerce_reg_to_size_sx(struct bpf_reg_state *reg, int size)
 	s64 init_s64_max, init_s64_min, s64_max, s64_min, u64_cval;
 	u64 top_smax_value, top_smin_value;
 	u64 num_bits = size * 8;
+	s64 limit = 1LL << (num_bits - 1);
+
+	/* If every value fits the signed N-bit range sign extension is a noop. */
+	if (reg_smin(reg) >= -limit && reg_smax(reg) < limit)
+		return;
+
+	/* See commen at reg_step_check_unsigned(). */
+	reg_step_reset(reg);
 
 	if (tnum_is_const(reg->var_off)) {
 		u64_cval = reg->var_off.value;
@@ -6020,6 +6176,16 @@ static void coerce_subreg_to_size_sx(struct bpf_reg_state *reg, int size)
 	u32 top_smax_value, top_smin_value;
 	u32 num_bits = size * 8;
 
+	/*
+	 * If value fits the non-negative half of the signed N-bit range,
+	 * sign extension is a noop.
+	 */
+	if (reg_umax(reg) < (1ULL << (num_bits - 1)))
+		return;
+
+	/* See comment at reg_step_check_unsigned(). */
+	reg_step_reset(reg);
+
 	if (tnum_is_const(reg->var_off)) {
 		u32_val = reg->var_off.value;
 		if (size == 1)
@@ -6729,6 +6895,7 @@ static void add_scalar_to_reg(struct bpf_reg_state *dst_reg, s64 val)
 	fake_reg.type = SCALAR_VALUE;
 	__mark_reg_known(&fake_reg, val);
 
+	scalar_step_add(dst_reg, dst_reg, &fake_reg);
 	scalar32_min_max_add(dst_reg, &fake_reg);
 	scalar_min_max_add(dst_reg, &fake_reg);
 	dst_reg->var_off = tnum_add(dst_reg->var_off, fake_reg.var_off);
@@ -15693,6 +15860,24 @@ static int sanitize_check_bounds(struct bpf_verifier_env *env,
 	return 0;
 }
 
+static void scalar_step_mul(struct bpf_reg_state *dst_reg, struct bpf_reg_state *src_reg)
+{
+	if (tnum_is_const(src_reg->var_off))
+		scalar_step_scale(dst_reg, src_reg->var_off.value);
+	else
+		reg_step_reset(dst_reg);
+}
+
+static void scalar_step_lsh(struct bpf_reg_state *dst_reg, struct bpf_reg_state *src_reg)
+{
+	u64 amount = src_reg->var_off.value;
+
+	if (tnum_is_const(src_reg->var_off) && amount < 64)
+		scalar_step_scale(dst_reg, 1ULL << amount);
+	else
+		reg_step_reset(dst_reg);
+}
+
 /* Handles arithmetic on a pointer and a scalar: computes new min/max and var_off.
  * Caller should also handle BPF_MOV case separately.
  * If we return -EACCES, caller may want to try again treating pointer as a
@@ -15861,6 +16046,7 @@ static int adjust_ptr_min_max_vals(struct bpf_verifier_env *env, struct bpf_insn
 		 * added into the variable offset, and we copy the fixed offset
 		 * from ptr_reg.
 		 */
+		scalar_step_add(dst_reg, ptr_reg, off_reg);
 		dst_reg->r64 = cnum64_add(ptr_reg->r64, off_reg->r64);
 		dst_reg->var_off = tnum_add(ptr_reg->var_off, off_reg->var_off);
 		dst_reg->raw = ptr_reg->raw;
@@ -15922,6 +16108,7 @@ static int adjust_ptr_min_max_vals(struct bpf_verifier_env *env, struct bpf_insn
 			if ((!known && smin_val < 0) || dst_reg->range < 0)
 				memset(&dst_reg->raw, 0, sizeof(dst_reg->raw));
 		}
+		reg_step_reset(dst_reg);
 		break;
 	case BPF_AND:
 	case BPF_OR:
@@ -16629,6 +16816,7 @@ static void scalar_byte_swap(struct bpf_reg_state *dst_reg, struct bpf_insn *ins
 		 * Bounds will be re-derived from the new tnum later.
 		 */
 		__mark_reg_unbounded(dst_reg);
+		reg_step_reset(dst_reg);
 	}
 	/* For bswap16/32, truncate dst register to match the swapped size */
 	if (insn->imm == 16 || insn->imm == 32)
@@ -16755,6 +16943,7 @@ static int adjust_scalar_min_max_vals(struct bpf_verifier_env *env,
 	 */
 	switch (opcode) {
 	case BPF_ADD:
+		scalar_step_add(dst_reg, dst_reg, &src_reg);
 		scalar32_min_max_add(dst_reg, &src_reg);
 		scalar_min_max_add(dst_reg, &src_reg);
 		dst_reg->var_off = tnum_add(dst_reg->var_off, src_reg.var_off);
@@ -16763,6 +16952,7 @@ static int adjust_scalar_min_max_vals(struct bpf_verifier_env *env,
 		scalar32_min_max_sub(dst_reg, &src_reg);
 		scalar_min_max_sub(dst_reg, &src_reg);
 		dst_reg->var_off = tnum_sub(dst_reg->var_off, src_reg.var_off);
+		reg_step_reset(dst_reg);
 		break;
 	case BPF_NEG:
 		env->fake_reg[0] = *dst_reg;
@@ -16770,8 +16960,10 @@ static int adjust_scalar_min_max_vals(struct bpf_verifier_env *env,
 		scalar32_min_max_sub(dst_reg, &env->fake_reg[0]);
 		scalar_min_max_sub(dst_reg, &env->fake_reg[0]);
 		dst_reg->var_off = tnum_neg(env->fake_reg[0].var_off);
+		reg_step_reset(dst_reg);
 		break;
 	case BPF_MUL:
+		scalar_step_mul(dst_reg, &src_reg);
 		dst_reg->var_off = tnum_mul(dst_reg->var_off, src_reg.var_off);
 		scalar32_min_max_mul(dst_reg, &src_reg);
 		scalar_min_max_mul(dst_reg, &src_reg);
@@ -16792,6 +16984,7 @@ static int adjust_scalar_min_max_vals(struct bpf_verifier_env *env,
 				scalar_min_max_sdiv(dst_reg, &src_reg);
 			else
 				scalar_min_max_udiv(dst_reg, &src_reg);
+		reg_step_reset(dst_reg);
 		break;
 	case BPF_MOD:
 		/* BPF mod specification: x % 0 = x */
@@ -16807,6 +17000,7 @@ static int adjust_scalar_min_max_vals(struct bpf_verifier_env *env,
 				scalar_min_max_smod(dst_reg, &src_reg);
 			else
 				scalar_min_max_umod(dst_reg, &src_reg);
+		reg_step_reset(dst_reg);
 		break;
 	case BPF_AND:
 		if (tnum_is_const(src_reg.var_off)) {
@@ -16817,6 +17011,7 @@ static int adjust_scalar_min_max_vals(struct bpf_verifier_env *env,
 		dst_reg->var_off = tnum_and(dst_reg->var_off, src_reg.var_off);
 		scalar32_min_max_and(dst_reg, &src_reg);
 		scalar_min_max_and(dst_reg, &src_reg);
+		reg_step_reset(dst_reg);
 		break;
 	case BPF_OR:
 		if (tnum_is_const(src_reg.var_off)) {
@@ -16827,13 +17022,16 @@ static int adjust_scalar_min_max_vals(struct bpf_verifier_env *env,
 		dst_reg->var_off = tnum_or(dst_reg->var_off, src_reg.var_off);
 		scalar32_min_max_or(dst_reg, &src_reg);
 		scalar_min_max_or(dst_reg, &src_reg);
+		reg_step_reset(dst_reg);
 		break;
 	case BPF_XOR:
 		dst_reg->var_off = tnum_xor(dst_reg->var_off, src_reg.var_off);
 		scalar32_min_max_xor(dst_reg, &src_reg);
 		scalar_min_max_xor(dst_reg, &src_reg);
+		reg_step_reset(dst_reg);
 		break;
 	case BPF_LSH:
+		scalar_step_lsh(dst_reg, &src_reg);
 		if (alu32)
 			scalar32_min_max_lsh(dst_reg, &src_reg);
 		else
@@ -16844,15 +17042,18 @@ static int adjust_scalar_min_max_vals(struct bpf_verifier_env *env,
 			scalar32_min_max_rsh(dst_reg, &src_reg);
 		else
 			scalar_min_max_rsh(dst_reg, &src_reg);
+		reg_step_reset(dst_reg);
 		break;
 	case BPF_ARSH:
 		if (alu32)
 			scalar32_min_max_arsh(dst_reg, &src_reg);
 		else
 			scalar_min_max_arsh(dst_reg, &src_reg);
+		reg_step_reset(dst_reg);
 		break;
 	case BPF_END:
 		scalar_byte_swap(dst_reg, insn);
+		reg_step_reset(dst_reg);
 		break;
 	default:
 		break;
@@ -17759,6 +17960,7 @@ static void regs_refine_cond_op(struct bpf_reg_state *reg1, struct bpf_reg_state
 		 * violations if we're on a dead branch.
 		 */
 		__mark_reg_unbounded(reg1);
+		reg_step_reset(reg1);
 		if (is_jmp32) {
 			t = tnum_and(tnum_subreg(reg1->var_off), tnum_const(~val));
 			reg1->var_off = tnum_with_subreg(reg1->var_off, t);
@@ -18081,6 +18283,7 @@ static void sync_linked_regs(struct bpf_verifier_env *env, struct bpf_verifier_s
 			reg->delta = saved_off;
 			reg->id = saved_id;
 
+			scalar_step_add(reg, reg, &fake_reg);
 			scalar32_min_max_add(reg, &fake_reg);
 			scalar_min_max_add(reg, &fake_reg);
 			reg->var_off = tnum_add(reg->var_off, fake_reg.var_off);

-- 
2.53.0

Re: [PATCH bpf-next v3 19/51] bpf: representation for intervals with steps

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 5607 bytes --]

> bpf: representation for intervals with steps

TL;DR:
- Commit message says eq/neq branches ignore base/step, but they appear to
  change branch pruning
- Comment in reg_bounds_sync() names reg_umin() where reg_smin() seems
  intended (nit)

>     bpf: representation for intervals with steps
>
>     Extend scalar register tracking with a linear "base + step * k" description:
>     each scalar carries two u16 fields, base and step (invariant: base < step),
>     recording that its value is some point on the line base + step * k.
>     A fresh scalar starts as base=0, step=1, i.e. any integer.
>
>     This lets the verifier reason about strided values, e.g. an index multiplied
>     by an element size, which neither the min/max bounds nor the tnum can capture
>     for a non-power-of-2 stride. Such reasoning is necessary to allow
>     array access with a variable offset through PTR_TO_BTF_ID, e.g.
>     for expressions like p->arr[i].field.
>
>     reg_bounds_sync() uses the description in two ways:
>     - __reg_bound_offset() clears the low bits of var_off implied by the number
>       of trailing zeros shared by base and step.
>     - __update_bounds_from_step() intersects the 64-bit range with the line via
>       cnum64_intersect_linear(), snapping smin/smax to actual points on it.
>       This is the last step of the sync, as refining the range from var_off
>       can move the bounds off the line.
>
>     range_within() is extended to check whether the cur register's equation
>     describes a subset of the points described by the old register's equation.
>
>     Scalar ALU operations update base/step as follows:
>     - ADD of a constant (including pointer + scalar): keep step, shift base;
>     - MUL by a constant: scale step and base by the constant;
>     - LSH by a constant: scale step and base by the corresponding power of two;
>     - sign extension: keep the register unchanged if all values fit the target
>       type, reset to base=0, step=1 otherwise;
>     - All other operations reset to base=0, step=1.
>
>     Branch (eq/neq) checks do not take into account base/step yet.
>
>     Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>

The changelog says "Branch (eq/neq) checks do not take into account base/step
yet."  Is that accurate for this patch?

is_scalar_branch_taken() ends by calling simulate_both_branches_taken().  That
function runs regs_refine_cond_op() and reg_bounds_sync() on copies of the
registers, and treats range_bounds_violation() as proof that a branch is dead.

reg_bounds_sync() now ends with __update_bounds_from_step(), and
cnum64_intersect_linear() returns CNUM64_EMPTY when the refined range holds no
point of base + step * k.

For example, take r1 in [0, 30] with step=3, base=0 (say from r0 *= 3 with r0
in [0, 10]), followed by:

    if r1 == 4 goto L

The tnum and min/max checks cannot decide this.  In
simulate_both_branches_taken() the true branch refines r64 to [4, 4], and then
in cnum64_intersect_linear():

    d = imod(base - imod(smin, step), step);   /* imod(0 - 1, 3) == 2 */
    if ((u64)smax - (u64)smin < d)              /* 0 < 2 */
            return CNUM64_EMPTY;

So the JEQ branch is treated as dead (pred = 0) purely because of
base/step.  JNE behaves the same way, and JNE against a range edge now snaps
the bound onto the line ([0, 29] becomes [0, 27]).

This looks sound as long as base/step are sound, but it is a real change to
branch pruning, and mark_chain_precision() is called on the compared registers
whenever pred >= 0.  Should the changelog describe this instead of saying
eq/neq branches ignore base/step?

> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index 47a632a6749a..f5b2500335a7 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c

[ ... ]

> @@ -2208,6 +2337,15 @@ static void reg_bounds_sync(struct bpf_reg_state *reg)
>  	 * then new var_off is (0; 0x7f...fc) which improves our umax.
>  	 */
>  	__update_reg_bounds(reg);
> +	/*
> +	 * Both tnum and base+step define a sequence of points,
> +	 * potentially w/o intersection, or intersecting after N
> +	 * tnum_step()'s, where N can be large. On the other hand,
> +	 * step+base *must* be in sync with r64 in order for
> +	 * array traversal logic starting at reg_umin() to be sound.
                                          ^^^^^^^^^^

This isn't a bug, but is reg_umin() the right function to name here?  The
array traversal logic isn't part of this patch.

It is added later in the series by b312df27df3b ("bpf: varying offset access
support for PTR_TO_BTF_ID pointers"), and that code starts from the signed
minimum rather than reg_umin():

    if (check_add_overflow(reg_smin(reg), (s64)_off, &min_off) || ...

cnum64_intersect_linear() also snaps the signed bounds (cnum64_smin() and
cnum64_smax()).  For ranges with negative values, reg_umin() is not on the
line at all.  For example, [-5, 7] with step=3, base=1 has umin 0, and
0 mod 3 != 1.

The claim that step+base "*must* be in sync with r64" also does not seem to
hold for signed-wrapping ranges, since cnum64_intersect_linear() returns those
unchanged when cnum64_srange_overflow() is true.

Could the comment refer to reg_smin() and the smin/smax snapping instead?

> +	 * Hence, place this as a final step in the reg_bounds_sync().
> +	 */
> +	__update_bounds_from_step(reg);
>  }
>


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/37923187511

Re: [PATCH bpf-next v3 19/51] bpf: representation for intervals with steps

[Eduard Zingerman]

On Fri, 2026-10-09 at 12:07 +0000, bot+bpf-ci@kernel.org wrote:
> > bpf: representation for intervals with steps
> 
> TL;DR:
> - Commit message says eq/neq branches ignore base/step, but they appear to
>   change branch pruning
> - Comment in reg_bounds_sync() names reg_umin() where reg_smin() seems
>   intended (nit)
> 
> >     bpf: representation for intervals with steps
> > 
> >     Extend scalar register tracking with a linear "base + step * k" description:
> >     each scalar carries two u16 fields, base and step (invariant: base < step),
> >     recording that its value is some point on the line base + step * k.
> >     A fresh scalar starts as base=0, step=1, i.e. any integer.
> > 
> >     This lets the verifier reason about strided values, e.g. an index multiplied
> >     by an element size, which neither the min/max bounds nor the tnum can capture
> >     for a non-power-of-2 stride. Such reasoning is necessary to allow
> >     array access with a variable offset through PTR_TO_BTF_ID, e.g.
> >     for expressions like p->arr[i].field.
> > 
> >     reg_bounds_sync() uses the description in two ways:
> >     - __reg_bound_offset() clears the low bits of var_off implied by the number
> >       of trailing zeros shared by base and step.
> >     - __update_bounds_from_step() intersects the 64-bit range with the line via
> >       cnum64_intersect_linear(), snapping smin/smax to actual points on it.
> >       This is the last step of the sync, as refining the range from var_off
> >       can move the bounds off the line.
> > 
> >     range_within() is extended to check whether the cur register's equation
> >     describes a subset of the points described by the old register's equation.
> > 
> >     Scalar ALU operations update base/step as follows:
> >     - ADD of a constant (including pointer + scalar): keep step, shift base;
> >     - MUL by a constant: scale step and base by the constant;
> >     - LSH by a constant: scale step and base by the corresponding power of two;
> >     - sign extension: keep the register unchanged if all values fit the target
> >       type, reset to base=0, step=1 otherwise;
> >     - All other operations reset to base=0, step=1.
> > 
> >     Branch (eq/neq) checks do not take into account base/step yet.
> > 
> >     Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
> 
> The changelog says "Branch (eq/neq) checks do not take into account base/step
> yet."  Is that accurate for this patch?
> 
> is_scalar_branch_taken() ends by calling simulate_both_branches_taken().  That
> function runs regs_refine_cond_op() and reg_bounds_sync() on copies of the
> registers, and treats range_bounds_violation() as proof that a branch is dead.
> 
> reg_bounds_sync() now ends with __update_bounds_from_step(), and
> cnum64_intersect_linear() returns CNUM64_EMPTY when the refined range holds no
> point of base + step * k.
> 
> For example, take r1 in [0, 30] with step=3, base=0 (say from r0 *= 3 with r0
> in [0, 10]), followed by:
> 
>     if r1 == 4 goto L
> 
> The tnum and min/max checks cannot decide this.  In
> simulate_both_branches_taken() the true branch refines r64 to [4, 4], and then
> in cnum64_intersect_linear():
> 
>     d = imod(base - imod(smin, step), step);   /* imod(0 - 1, 3) == 2 */
>     if ((u64)smax - (u64)smin < d)              /* 0 < 2 */
>             return CNUM64_EMPTY;
> 
> So the JEQ branch is treated as dead (pred = 0) purely because of
> base/step.  JNE behaves the same way, and JNE against a range edge now snaps
> the bound onto the line ([0, 29] becomes [0, 27]).
> 
> This looks sound as long as base/step are sound, but it is a real change to
> branch pruning, and mark_chain_precision() is called on the compared registers
> whenever pred >= 0.  Should the changelog describe this instead of saying
> eq/neq branches ignore base/step?

Kind of. Sort of.
There is no explicit handling in is_scalar_branch_taken(),
as well as it is not wired for pointers.
I didn't think through if simulate_both_branches_taken()
is sufficient or some explicit rules is_scalar_branch_taken()
might be of use as well.

> 
> > diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> > index 47a632a6749a..f5b2500335a7 100644
> > --- a/kernel/bpf/verifier.c
> > +++ b/kernel/bpf/verifier.c
> 
> [ ... ]
> 
> > @@ -2208,6 +2337,15 @@ static void reg_bounds_sync(struct bpf_reg_state *reg)
> >  	 * then new var_off is (0; 0x7f...fc) which improves our umax.
> >  	 */
> >  	__update_reg_bounds(reg);
> > +	/*
> > +	 * Both tnum and base+step define a sequence of points,
> > +	 * potentially w/o intersection, or intersecting after N
> > +	 * tnum_step()'s, where N can be large. On the other hand,
> > +	 * step+base *must* be in sync with r64 in order for
> > +	 * array traversal logic starting at reg_umin() to be sound.
>                                           ^^^^^^^^^^
> 
> This isn't a bug, but is reg_umin() the right function to name here?  The
> array traversal logic isn't part of this patch.

Yes, this should be reg_smin().

[PATCH bpf-next v3 20/51] bpf: varying offset access support for PTR_TO_BTF_ID pointers

[Eduard Zingerman]

Relax the requirements for reads like `p->arr[i].field`, where `p` is a
PTR_TO_BTF_ID, by allowing `i` to be a non-constant value.
This is useful for loops over arrays accessed from a PTR_TO_BTF_ID
when the verifier widens the loop induction variable.

Check that the access stays within array bounds based on the access
register's min and max bounds. Check that the access is aligned with
array elements using the register's `base` and `step`.

Technically, modify btf_struct_access() such that:
- it identifies `field` by walking BTF from the minimum offset
  (the register's reg_smin + static instruction offset);
- it remembers which arrays contain `field`;
- it checks that offsets `k * step` stay within the array
  and point to the same `field`;
- it does not check upper bounds for flexible arrays
  (as is already the case for constant-offset access).

Support for flexible arrays nested inside trailing struct members
(e.g. p->nested.data[i], with data declared as int data[]) is left
for future work.

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 kernel/bpf/btf.c      | 159 ++++++++++++++++++++++++++++++++++++++++++--------
 kernel/bpf/states.c   |   1 +
 kernel/bpf/verifier.c |  38 +++++++-----
 3 files changed, 159 insertions(+), 39 deletions(-)

diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index 0630675377aa..f4d119f3d5ef 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -2079,7 +2079,8 @@ static const struct resolve_vertex *env_stack_peak(struct btf_verifier_env *env)
  * *elem_id: id of u32
  * *total_nelems: (x * y).  Hence, individual elem size is
  *                (*type_size / *total_nelems)
- * *type_id: id of type if it's changed within the function, 0 if not
+ * *type_id: ID of the returned type if leading modifiers were stripped;
+ *           otherwise left unchanged.
  *
  * type: is not an array (e.g. const struct X)
  * return type: type "struct X"
@@ -2087,7 +2088,8 @@ static const struct resolve_vertex *env_stack_peak(struct btf_verifier_env *env)
  * *elem_type: same as return type ("struct X")
  * *elem_id: 0
  * *total_nelems: 1
- * *type_id: id of type if it's changed within the function, 0 if not
+ * *type_id: ID of the returned type if leading modifiers were stripped;
+ *           otherwise left unchanged.
  */
 static const struct btf_type *
 __btf_resolve_size(const struct btf *btf, const struct btf_type *type,
@@ -2120,7 +2122,8 @@ __btf_resolve_size(const struct btf *btf, const struct btf_type *type,
 		case BTF_KIND_CONST:
 		case BTF_KIND_RESTRICT:
 		case BTF_KIND_TYPE_TAG:
-			id = type->type;
+			if (!array_type)
+				id = type->type;
 			type = btf_type_by_id(btf, type->type);
 			break;
 
@@ -7549,6 +7552,14 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type,
 }
 EXPORT_SYMBOL_GPL(btf_ctx_access);
 
+#define MAX_ARRAYS_WALK 16
+
+/* Description of an array crossed while walking a BTF access chain */
+struct array_access {
+	u32 id;		/* BTF_KIND_ARRAY type id */
+	u32 off;	/* offset of the access within the array */
+};
+
 enum bpf_struct_walk_result {
 	/* < 0 error */
 	WALK_SCALAR = 0,
@@ -7557,10 +7568,43 @@ enum bpf_struct_walk_result {
 	WALK_STRUCT,
 };
 
+static const struct btf_member *find_flex_member(const struct btf *btf, const struct btf_type *t)
+{
+	const struct btf_array *array;
+	const struct btf_type *mtype;
+	const struct btf_member *member;
+	u32 vlen = btf_type_vlen(t);
+
+	if (vlen == 0)
+		return NULL;
+
+	member = btf_type_member(t) + vlen - 1;
+	mtype = btf_type_skip_modifiers(btf, member->type, NULL);
+	if (!btf_type_is_array(mtype))
+		return NULL;
+
+	array = (const struct btf_array *)(mtype + 1);
+	if (array->nelems != 0)
+		return NULL;
+
+	return member;
+}
+
+static int push_array(struct array_access *arrays, u32 *arrays_cnt, struct array_access item)
+{
+	if (arrays == NULL)
+		return 0;
+	if (*arrays_cnt == MAX_ARRAYS_WALK)
+		return -1;
+	arrays[(*arrays_cnt)++] = item;
+	return 0;
+}
+
 static int btf_struct_walk(struct bpf_verifier_log *log, const struct btf *btf,
 			   const struct btf_type *t, int off, int size,
 			   u32 *next_btf_id, enum bpf_type_flag *flag,
-			   const char **field_name, bool walk_flex_arrays)
+			   const char **field_name, bool walk_flex_arrays,
+			   struct array_access *arrays, u32 *arrays_cnt)
 {
 	u32 i, moff, mtrue_end, msize = 0, total_nelems = 0;
 	const struct btf_type *mtype, *elem_type = NULL;
@@ -7595,22 +7639,19 @@ static int btf_struct_walk(struct bpf_verifier_log *log, const struct btf *btf,
 		/* If the last element is a variable size array, we may
 		 * need to relax the rule.
 		 */
-		if (vlen == 0)
+		member = find_flex_member(btf, t);
+		if (!member)
 			goto error;
 
-		member = btf_type_member(t) + vlen - 1;
-		mtype = btf_type_skip_modifiers(btf, member->type,
-						NULL);
-		if (!btf_type_is_array(mtype))
+		moff = __btf_member_bit_offset(t, member) / 8;
+		if (off < moff)
 			goto error;
 
+		mtype = btf_type_skip_modifiers(btf, member->type, &mid);
 		array_elem = (struct btf_array *)(mtype + 1);
-		if (array_elem->nelems != 0)
-			goto error;
 
-		moff = __btf_member_bit_offset(t, member) / 8;
-		if (off < moff)
-			goto error;
+		if (push_array(arrays, arrays_cnt, (struct array_access){ mid, off - moff }))
+			return -E2BIG;
 
 		/* allow structure and integer */
 		t = btf_type_skip_modifiers(btf, array_elem->type,
@@ -7733,6 +7774,9 @@ static int btf_struct_walk(struct bpf_verifier_log *log, const struct btf *btf,
 			 *      the array's element as long as it is
 			 *      within the mtrue_end boundary.
 			 */
+			if (push_array(arrays, arrays_cnt,
+				       (struct array_access){ mid, off - moff }))
+				return -E2BIG;
 
 			/* skip empty array */
 			if (moff == mtrue_end)
@@ -7825,15 +7869,29 @@ static int btf_struct_walk(struct bpf_verifier_log *log, const struct btf *btf,
 
 int btf_struct_access(struct bpf_verifier_log *log,
 		      const struct bpf_reg_state *reg,
-		      int off, int size, enum bpf_access_type atype __maybe_unused,
+		      int _off, int size, enum bpf_access_type atype __maybe_unused,
 		      u32 *next_btf_id, enum bpf_type_flag *flag,
 		      const char **field_name)
 {
 	const struct btf *btf = reg->btf;
 	enum bpf_type_flag tmp_flag = 0;
+	struct array_access arrays[MAX_ARRAYS_WALK];
+	const struct btf_member *member;
 	const struct btf_type *t;
+	u32 i, arrays_cnt = 0;
 	u32 id = reg->btf_id;
-	int err;
+	s64 min_off, max_off, flex_off;
+	int err, off, ret;
+
+	if (check_add_overflow(reg_smin(reg), (s64)_off, &min_off) ||
+	    check_add_overflow(reg_smax(reg), (s64)_off, &max_off) ||
+	    min_off != (int)min_off) {
+		bpf_log(log,
+			"offset computation overflows: register offset range is [%lld, %lld], instruction offset is %d\n",
+			reg_smin(reg), reg_smax(reg), _off);
+		return -EINVAL;
+	}
+	off = min_off;
 
 	while (type_is_alloc(reg->type)) {
 		struct btf_struct_meta *meta;
@@ -7860,26 +7918,30 @@ int btf_struct_access(struct bpf_verifier_log *log,
 	t = btf_type_by_id(btf, id);
 	do {
 		err = btf_struct_walk(log, btf, t, off, size, &id, &tmp_flag,
-				      field_name, !type_is_alloc(reg->type));
-
+				      field_name, !type_is_alloc(reg->type), arrays, &arrays_cnt);
 		switch (err) {
 		case WALK_PTR:
 			/* For local types, the destination register cannot
 			 * become a pointer again.
 			 */
-			if (type_is_alloc(reg->type))
-				return SCALAR_VALUE;
+			if (type_is_alloc(reg->type)) {
+				ret = SCALAR_VALUE;
+				goto check_variable_offset;
+			}
 			/* If we found the pointer or scalar on t+off,
 			 * we're done.
 			 */
 			*next_btf_id = id;
 			*flag = tmp_flag;
-			return PTR_TO_BTF_ID;
+			ret = PTR_TO_BTF_ID;
+			goto check_variable_offset;
 		case WALK_PTR_UNTRUSTED:
 			*flag = MEM_RDONLY | PTR_UNTRUSTED;
-			return PTR_TO_MEM;
+			ret = PTR_TO_MEM;
+			goto check_variable_offset;
 		case WALK_SCALAR:
-			return SCALAR_VALUE;
+			ret = SCALAR_VALUE;
+			goto check_variable_offset;
 		case WALK_STRUCT:
 			/* We found nested struct, so continue the search
 			 * by diving in it. At this point the offset is
@@ -7899,6 +7961,55 @@ int btf_struct_access(struct bpf_verifier_log *log,
 	} while (t);
 
 	return -EINVAL;
+
+check_variable_offset:
+	if (min_off == max_off)
+		return ret;
+
+	/* Find an offset at which access would go to a flexible array tail (if any). */
+	t = btf_type_skip_modifiers(btf, reg->btf_id, NULL);
+	member = find_flex_member(btf, t);
+	flex_off = member ? __btf_member_bit_offset(t, member) / 8 : S64_MAX;
+
+	/*
+	 * If this is a varying offset access, the step recorded within a register
+	 * should correspond to one of the arrays visited while walking.
+	 */
+	for (i = 0; i < arrays_cnt; i++) {
+		s64 array_start, array_end, access_end;
+		u32 asize, esize, elem_id;
+
+		/*
+		 * 'arrays' records top-level arrays only, use __btf_resolve_size()
+		 * to get flattened representation.
+		 */
+		t = btf_type_by_id(btf, arrays[i].id);
+		if (IS_ERR(__btf_resolve_size(btf, t, &asize, NULL, &elem_id, NULL, NULL)) ||
+		    IS_ERR(btf_resolve_size(btf, btf_type_by_id(btf, elem_id), &esize)))
+			continue;
+		/* Make sure every accessed offset lands on the same field of some element. */
+		if (esize == 0 || reg->step % esize != 0)
+			continue;
+
+		array_start = min_off - arrays[i].off;
+		/* An array within the flexible tail has no upper bound to exceed */
+		if (array_start >= flex_off && btf_type_is_array(t) &&
+		    btf_type_array(t)->nelems == 0)
+			return ret;
+
+		/* the 'size' bytes read at max_off must stay within the array */
+		if (check_add_overflow(array_start, (s64)asize, &array_end) ||
+		    check_add_overflow(max_off, (s64)size, &access_end) ||
+		    access_end > array_end)
+			continue;
+		return ret;
+	}
+
+	bpf_log(log,
+		"invalid variable offset access into struct %s: offsets [%lld, %lld] size %d step %d do not match any array\n",
+		btf_name_by_offset(btf, btf_type_by_id(btf, reg->btf_id)->name_off),
+		min_off, max_off, size, reg->step);
+	return -EINVAL;
 }
 
 /* Check that two BTF types, each specified as an BTF object + id, are exactly
@@ -7939,7 +8050,7 @@ bool btf_struct_ids_match(struct bpf_verifier_log *log,
 	if (!type)
 		return false;
 	err = btf_struct_walk(log, btf, type, off, 1, &id, &flag, NULL,
-			      walk_flex_arrays);
+			      walk_flex_arrays, NULL, NULL);
 	if (err != WALK_STRUCT)
 		return false;
 
diff --git a/kernel/bpf/states.c b/kernel/bpf/states.c
index 981328f5eb60..2f6a7620eb72 100644
--- a/kernel/bpf/states.c
+++ b/kernel/bpf/states.c
@@ -633,6 +633,7 @@ static bool regsafe(struct bpf_verifier_env *env, struct bpf_reg_state *rold,
 	case PTR_TO_BUF:
 	case PTR_TO_TP_BUFFER:
 	case PTR_TO_STACK:
+	case PTR_TO_BTF_ID:
 		/* If the new min/max/var_off satisfy the old ones and
 		 * everything else matches, we are OK.
 		 */
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index f5b2500335a7..192efb722997 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -6554,6 +6554,7 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env,
 	const char *field_name = NULL;
 	enum bpf_type_flag flag = 0;
 	u32 btf_id = 0;
+	s64 min_off;
 	int ret;
 
 	if (!env->allow_ptr_leaks) {
@@ -6569,36 +6570,31 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env,
 		return -EINVAL;
 	}
 
-	if (!tnum_is_const(reg->var_off)) {
-		char tn_buf[48];
-
-		tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off);
+	if (check_add_overflow(reg_smin(reg), off, &min_off)) {
 		verbose(env,
-			"%s is ptr_%s invalid variable offset: off=%d, var_off=%s\n",
-			reg_arg_name(env, argno), tname, off, tn_buf);
+			"%s is ptr_%s access, offset computation overflows: register's minimal offset is %lld, instruction offset is %d\n",
+			reg_arg_name(env, argno), tname, reg_smin(reg), off);
 		return -EACCES;
 	}
 
-	off += reg->var_off.value;
-
-	if (off < 0) {
+	if (min_off < 0) {
 		verbose(env,
-			"%s is ptr_%s invalid negative access: off=%d\n",
-			reg_arg_name(env, argno), tname, off);
+			"%s is ptr_%s invalid negative access: off=%lld\n",
+			reg_arg_name(env, argno), tname, min_off);
 		return -EACCES;
 	}
 
 	if (reg->type & MEM_USER) {
 		verbose(env,
-			"%s is ptr_%s access user memory: off=%d\n",
-			reg_arg_name(env, argno), tname, off);
+			"%s is ptr_%s access user memory\n",
+			reg_arg_name(env, argno), tname);
 		return -EACCES;
 	}
 
 	if (reg->type & MEM_PERCPU) {
 		verbose(env,
-			"%s is ptr_%s access percpu memory: off=%d\n",
-			reg_arg_name(env, argno), tname, off);
+			"%s is ptr_%s access percpu memory\n",
+			reg_arg_name(env, argno), tname);
 		return -EACCES;
 	}
 
@@ -6608,6 +6604,18 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env,
 	}
 
 	if (env->ops->btf_struct_access && !type_is_alloc(reg->type) && atype == BPF_WRITE) {
+		if (!tnum_is_const(reg->var_off)) {
+			char tn_buf[48];
+
+			tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off);
+			verbose(env,
+				"%s is ptr_%s invalid variable offset: off=%d, var_off=%s\n",
+				reg_arg_name(env, argno), tname, off, tn_buf);
+			return -EACCES;
+		}
+
+		off += reg->var_off.value;
+
 		if (!btf_is_kernel(reg->btf)) {
 			verifier_bug(env, "reg->btf must be kernel btf");
 			return -EFAULT;

-- 
2.53.0

Re: [PATCH bpf-next v3 20/51] bpf: varying offset access support for PTR_TO_BTF_ID pointers

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] bpf: truncation of 64-bit var_off to 32-bit bypasses custom btf_struct_access bounds
- [High] bpf: unbounded variable offset read on trusted PTR_TO_BTF_ID bypasses PROBE_MEM
--

commit ec6d1e8a8c55dc442e46ac3d70d36a00ec8a9ffd
Author: Eduard Zingerman <eddyz87@gmail.com>

bpf: varying offset access support for PTR_TO_BTF_ID pointers

This commit relaxes requirements for varying offset accesses to PTR_TO_BTF_ID
pointers, verifying they stay within array bounds and align with elements.
It modifies btf_struct_access to validate minimum offsets and array spans,
while skipping upper bounds checks for flexible arrays.

> diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
> index 0630675377aaa..f4d119f3d5ef9 100644
> --- a/kernel/bpf/btf.c
> +++ b/kernel/bpf/btf.c
[ ... ]
> @@ -7899,6 +7961,55 @@ int btf_struct_access(struct bpf_verifier_log *log,
[ ... ]
> +		array_start = min_off - arrays[i].off;
> +		/* An array within the flexible tail has no upper bound to exceed */
> +		if (array_start >= flex_off && btf_type_is_array(t) &&
> +		    btf_type_array(t)->nelems == 0)
> +			return ret;

[Severity: High]
Does this allow unbounded variable offset reads on trusted pointers?

This logic explicitly skips the upper bound check for flexible arrays,
assuming PROBE_MEM will safely catch out-of-bounds page faults at runtime.

However, if the pointer being accessed is trusted (meaning it has the
PTR_TRUSTED flag and lacks PTR_UNTRUSTED), wouldn't bpf_may_fault_on_deref()
evaluate to false?

If bpf_may_fault_on_deref() is false, bpf_convert_ctx_accesses() won't
convert the load into a PROBE_MEM exception-handled load. Passing an
arbitrarily large offset here could result in a native load from unmapped
memory, potentially crashing the kernel.

> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index f5b2500335a7b..192efb7229975 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
[ ... ]
> @@ -6608,6 +6604,18 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env,
>  	}
>  
>  	if (env->ops->btf_struct_access && !type_is_alloc(reg->type) && atype == BPF_WRITE) {
> +		if (!tnum_is_const(reg->var_off)) {
> +			char tn_buf[48];
> +
> +			tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off);
> +			verbose(env,
> +				"%s is ptr_%s invalid variable offset: off=%d, var_off=%s\n",
> +				reg_arg_name(env, argno), tname, off, tn_buf);
> +			return -EACCES;
> +		}
> +
> +		off += reg->var_off.value;
> +
>  		if (!btf_is_kernel(reg->btf)) {
>  			verifier_bug(env, "reg->btf must be kernel btf");
>  			return -EFAULT;

[Severity: High]
Could this assignment truncate a large 64-bit value into a negative 32-bit
offset?

Earlier in check_ptr_to_btf_access(), the negative check for off was removed
and replaced with min_off. Now that off += reg->var_off.value is executed
down here for BPF_WRITE with custom handlers, a very large 64-bit
var_off.value could wrap into a negative 32-bit integer.

Since there is no negativity check immediately following this addition, the
truncated negative offset gets passed directly into the custom
btf_struct_access handler.

If a handler like bpf_scx_btf_struct_access compares this negative offset
against size_t boundaries, couldn't this bypass the bounds check and
allow out-of-bounds writes to kernel objects?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261009-scev-minimal-rebase-v3-0-bcba9bc53ac9@gmail.com?part=20

Re: [PATCH bpf-next v3 20/51] bpf: varying offset access support for PTR_TO_BTF_ID pointers

[Eduard Zingerman]

On Fri, 2026-10-09 at 11:41 +0000, sashiko-bot@kernel.org wrote:
> Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
> - [High] bpf: truncation of 64-bit var_off to 32-bit bypasses custom btf_struct_access bounds
> - [High] bpf: unbounded variable offset read on trusted PTR_TO_BTF_ID bypasses PROBE_MEM
> --
> 
> commit ec6d1e8a8c55dc442e46ac3d70d36a00ec8a9ffd
> Author: Eduard Zingerman <eddyz87@gmail.com>
> 
> bpf: varying offset access support for PTR_TO_BTF_ID pointers
> 
> This commit relaxes requirements for varying offset accesses to PTR_TO_BTF_ID
> pointers, verifying they stay within array bounds and align with elements.
> It modifies btf_struct_access to validate minimum offsets and array spans,
> while skipping upper bounds checks for flexible arrays.
> 
> > diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
> > index 0630675377aaa..f4d119f3d5ef9 100644
> > --- a/kernel/bpf/btf.c
> > +++ b/kernel/bpf/btf.c
> [ ... ]
> > @@ -7899,6 +7961,55 @@ int btf_struct_access(struct bpf_verifier_log *log,
> [ ... ]
> > +		array_start = min_off - arrays[i].off;
> > +		/* An array within the flexible tail has no upper bound to exceed */
> > +		if (array_start >= flex_off && btf_type_is_array(t) &&
> > +		    btf_type_array(t)->nelems == 0)
> > +			return ret;
> 
> [Severity: High]
> Does this allow unbounded variable offset reads on trusted pointers?
> 
> This logic explicitly skips the upper bound check for flexible arrays,
> assuming PROBE_MEM will safely catch out-of-bounds page faults at runtime.
> 
> However, if the pointer being accessed is trusted (meaning it has the
> PTR_TRUSTED flag and lacks PTR_UNTRUSTED), wouldn't bpf_may_fault_on_deref()
> evaluate to false?
> 
> If bpf_may_fault_on_deref() is false, bpf_convert_ctx_accesses() won't
> convert the load into a PROBE_MEM exception-handled load. Passing an
> arbitrarily large offset here could result in a native load from unmapped
> memory, potentially crashing the kernel.

This is a pre-existing bug.
I think the correct fix here is to always treat flexible array tails
access as un-trusted, varying offset or not.

> 
> > diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> > index f5b2500335a7b..192efb7229975 100644
> > --- a/kernel/bpf/verifier.c
> > +++ b/kernel/bpf/verifier.c
> [ ... ]
> > @@ -6608,6 +6604,18 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env,
> >  	}
> >  
> >  	if (env->ops->btf_struct_access && !type_is_alloc(reg->type) && atype == BPF_WRITE) {
> > +		if (!tnum_is_const(reg->var_off)) {
> > +			char tn_buf[48];
> > +
> > +			tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off);
> > +			verbose(env,
> > +				"%s is ptr_%s invalid variable offset: off=%d, var_off=%s\n",
> > +				reg_arg_name(env, argno), tname, off, tn_buf);
> > +			return -EACCES;
> > +		}
> > +
> > +		off += reg->var_off.value;
> > +
> >  		if (!btf_is_kernel(reg->btf)) {
> >  			verifier_bug(env, "reg->btf must be kernel btf");
> >  			return -EFAULT;
> 
> [Severity: High]
> Could this assignment truncate a large 64-bit value into a negative 32-bit
> offset?
> 
> Earlier in check_ptr_to_btf_access(), the negative check for off was removed
> and replaced with min_off. Now that off += reg->var_off.value is executed
> down here for BPF_WRITE with custom handlers, a very large 64-bit
> var_off.value could wrap into a negative 32-bit integer.
> 
> Since there is no negativity check immediately following this addition, the
> truncated negative offset gets passed directly into the custom
> btf_struct_access handler.
> 
> If a handler like bpf_scx_btf_struct_access compares this negative offset
> against size_t boundaries, couldn't this bypass the bounds check and
> allow out-of-bounds writes to kernel objects?

I don't think this can happen.
reg->var_off.value is bound by BPF_MAX_VAR_OFF.
`off` comes from 16-bit instruction offset.
Plus there is a check:

  check_add_overflow(reg_smin(reg), off, &min_off)

just before.

[PATCH bpf-next v3 21/51] bpf: save DFS postorder numbers for program instructions

[Eduard Zingerman]

Dominator intersections and SCEV worklist scheduling need to compare
instructions by their DFS postorder rank. Save per-instruction
postorder numbers alongside the existing postorder sequence.

This commit modifies bpf_compute_postorder() to actually do the DFS
traversal, instead of scheduling traversal of all siblings at once.

Consider a graph:

  A -> B
  A -> C
  C -> B

Old result: C, B, A
DFS result: B, C, A

Fixes: efcda22aa541 ("bpf: compute instructions postorder per subprogram")
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 include/linux/bpf_verifier.h |  1 +
 kernel/bpf/cfg.c             | 56 +++++++++++++++++++++++++++-----------------
 kernel/bpf/const_fold.c      |  2 ++
 kernel/bpf/verifier.c        |  1 +
 4 files changed, 39 insertions(+), 21 deletions(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 2560754ed56c..bd7733fbe930 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1036,6 +1036,7 @@ struct bpf_verifier_env {
 		 * see bpf_subprog_info->postorder_start.
 		 */
 		int *insn_postorder;
+		int *postorder_nums;
 		int cur_stack;
 		/* current position in the insn_postorder vector */
 		int cur_postorder;
diff --git a/kernel/bpf/cfg.c b/kernel/bpf/cfg.c
index d8a579680e5b..bce7b5abcfda 100644
--- a/kernel/bpf/cfg.c
+++ b/kernel/bpf/cfg.c
@@ -767,6 +767,11 @@ int bpf_check_cfg(struct bpf_verifier_env *env)
 	return ret;
 }
 
+struct dfs_state {
+	u32 traversed:1;
+	u32 next_succ:31;
+};
+
 /*
  * For each subprogram 'i' fill array env->cfg.insn_subprogram sub-range
  * [env->subprog_info[i].postorder_start, env->subprog_info[i+1].postorder_start)
@@ -774,43 +779,52 @@ int bpf_check_cfg(struct bpf_verifier_env *env)
  */
 int bpf_compute_postorder(struct bpf_verifier_env *env)
 {
-	u32 cur_postorder, i, top, stack_sz, s;
-	int *stack = NULL, *postorder = NULL, *state = NULL;
-	struct bpf_iarray *succ;
+	int *stack = NULL, *postorder = NULL, *postorder_nums = NULL;
+	int subprog_idx, stack_sz, cur, s, cur_postorder, start;
+	struct dfs_state *state = NULL;
+	struct bpf_iarray *succ = NULL;
 
+	postorder_nums = kvzalloc_objs(int, env->prog->len, GFP_KERNEL_ACCOUNT);
 	postorder = kvzalloc_objs(int, env->prog->len, GFP_KERNEL_ACCOUNT);
-	state = kvzalloc_objs(int, env->prog->len, GFP_KERNEL_ACCOUNT);
 	stack = kvzalloc_objs(int, env->prog->len, GFP_KERNEL_ACCOUNT);
-	if (!postorder || !state || !stack) {
+	state = kvzalloc_objs(struct dfs_state, env->prog->len, GFP_KERNEL_ACCOUNT);
+	if (!postorder_nums || !postorder || !stack || !state) {
+		kvfree(postorder_nums);
 		kvfree(postorder);
-		kvfree(state);
 		kvfree(stack);
+		kvfree(state);
 		return -ENOMEM;
 	}
 	cur_postorder = 0;
-	for (i = 0; i < env->subprog_cnt; i++) {
-		env->subprog_info[i].postorder_start = cur_postorder;
-		stack[0] = env->subprog_info[i].start;
+	for (subprog_idx = 0; subprog_idx < env->subprog_cnt; subprog_idx++) {
+		start = env->subprog_info[subprog_idx].start;
+		env->subprog_info[subprog_idx].postorder_start = cur_postorder;
+		stack[0] = start;
 		stack_sz = 1;
+		state[start].traversed = true;
 		do {
-			top = stack[stack_sz - 1];
-			state[top] |= DISCOVERED;
-			if (state[top] & EXPLORED) {
-				postorder[cur_postorder++] = top;
+			cur = stack[stack_sz - 1];
+			succ = bpf_insn_successors(env, cur);
+			if (state[cur].next_succ == succ->cnt) {
+				postorder_nums[cur] = cur_postorder;
+				postorder[cur_postorder] = cur;
+				cur_postorder++;
 				stack_sz--;
 				continue;
 			}
-			succ = bpf_insn_successors(env, top);
-			for (s = 0; s < succ->cnt; ++s) {
-				if (!state[succ->items[s]]) {
-					stack[stack_sz++] = succ->items[s];
-					state[succ->items[s]] |= DISCOVERED;
-				}
+			s = succ->items[state[cur].next_succ];
+			if (!state[s].traversed) {
+				state[s].traversed = true;
+				state[s].next_succ = 0;
+				stack[stack_sz] = s;
+				stack_sz++;
+				continue;
 			}
-			state[top] |= EXPLORED;
+			state[cur].next_succ++;
 		} while (stack_sz);
 	}
-	env->subprog_info[i].postorder_start = cur_postorder;
+	env->subprog_info[subprog_idx].postorder_start = cur_postorder;
+	env->cfg.postorder_nums = postorder_nums;
 	env->cfg.insn_postorder = postorder;
 	env->cfg.cur_postorder = cur_postorder;
 	kvfree(stack);
diff --git a/kernel/bpf/const_fold.c b/kernel/bpf/const_fold.c
index b1528adbeb79..8dd8c5958eb8 100644
--- a/kernel/bpf/const_fold.c
+++ b/kernel/bpf/const_fold.c
@@ -398,5 +398,7 @@ int bpf_prune_dead_branches(struct bpf_verifier_env *env)
 	/* recompute postorder, since CFG has changed */
 	kvfree(env->cfg.insn_postorder);
 	env->cfg.insn_postorder = NULL;
+	kvfree(env->cfg.postorder_nums);
+	env->cfg.postorder_nums = NULL;
 	return bpf_compute_postorder(env);
 }
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 192efb722997..91f114947b8b 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -23221,6 +23221,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	vfree(env->insn_aux_data);
 	kvfree(env->fd_array);
 	bpf_stack_liveness_free(env);
+	kvfree(env->cfg.postorder_nums);
 	kvfree(env->cfg.insn_postorder);
 	kvfree(env->scc_info);
 	kvfree(env->succ);

-- 
2.53.0

[PATCH bpf-next v3 22/51] bpf: move the live-register and SCC printout to a standalone function

[Eduard Zingerman]

Tests for multiple analyses performed by the verifier need the
verifier log to contain analysis results alongside the program
disassembly. This patch moves the log-level-2 program dump from
bpf_compute_live_registers() to a standalone function called from
bpf_check(), in order to provide a common logging function for such
analyses (and thus avoid printing program disassembly multiple times).

Rename the dump header from "Live regs before insn:" to
"Program dump (scc? insn#: live_regs_before):" and print available
source-line information before the corresponding instructions.

Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 kernel/bpf/liveness.c                              | 28 +---------------
 kernel/bpf/verifier.c                              | 37 ++++++++++++++++++++++
 .../selftests/bpf/progs/verifier_live_stack.c      |  2 +-
 3 files changed, 39 insertions(+), 28 deletions(-)

diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index 8d8428ccc505..e657e6d95408 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -2664,8 +2664,7 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
 	struct bpf_insn *insns = env->prog->insnsi;
 	struct insn_live_regs *state;
 	int insn_cnt = env->prog->len;
-	u64 pos, insn_pos;
-	int err = 0, i, j, subprog, start, end;
+	int err = 0, i, subprog, start, end;
 	bool changed, ret_reg_pair;
 
 	/* Use the following algorithm:
@@ -2754,31 +2753,6 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
 		insn_aux[i].zext_dst = def32 >= 0 && (mask_hi(out) & BIT(def32));
 	}
 
-	if (env->log.level & BPF_LOG_LEVEL2) {
-		verbose(env, "Live regs before insn:\n");
-		for (i = 0; i < insn_cnt; ++i) {
-			if (env->insn_aux_data[i].scc)
-				verbose(env, "%3d ", env->insn_aux_data[i].scc);
-			else
-				verbose(env, "    ");
-			verbose(env, "%3d: ", i);
-			for (j = BPF_REG_0; j < BPF_REG_10; ++j)
-				if (insn_aux[i].live_regs_before & BIT(j))
-					verbose(env, "%d", j);
-				else
-					verbose(env, ".");
-			verbose(env, " ");
-			pos = env->log.end_pos;
-			bpf_verbose_insn(env, &insns[i]);
-			insn_pos = env->log.end_pos;
-			if (insn_aux[i].zext_dst)
-				verbose(env, "%*c; zext", bpf_vlog_alignment(insn_pos - pos), ' ');
-			verbose(env, "\n");
-			if (bpf_is_ldimm64(&insns[i]))
-				i++;
-		}
-	}
-
 out:
 	kvfree(state);
 	return err;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 91f114947b8b..9260ee290072 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -22863,6 +22863,40 @@ static bool bpf_prog_reenters_datapath(const struct bpf_prog *prog)
 	return false;
 }
 
+/* Various log level 2 information about the program */
+static void log_program(struct bpf_verifier_env *env)
+{
+	struct bpf_insn_aux_data *insn_aux = env->insn_aux_data;
+	struct bpf_insn *insns = env->prog->insnsi;
+	u32 insn_cnt = env->prog->len;
+	u64 pos, insn_pos;
+	u32 i, j;
+
+	verbose(env, "Program dump (scc? insn#: live_regs_before):\n");
+	for (i = 0; i < insn_cnt; ++i) {
+		verbose_linfo(env, i, "    ; ");
+		if (env->insn_aux_data[i].scc)
+			verbose(env, "%3d ", env->insn_aux_data[i].scc);
+		else
+			verbose(env, "    ");
+		verbose(env, "%3d: ", i);
+		for (j = BPF_REG_0; j < BPF_REG_10; ++j)
+			if (insn_aux[i].live_regs_before & BIT(j))
+				verbose(env, "%d", j);
+			else
+				verbose(env, ".");
+		verbose(env, " ");
+		pos = env->log.end_pos;
+		bpf_verbose_insn(env, &insns[i]);
+		insn_pos = env->log.end_pos;
+		if (insn_aux[i].zext_dst)
+			verbose(env, "%*c; zext", bpf_vlog_alignment(insn_pos - pos), ' ');
+		verbose(env, "\n");
+		if (bpf_is_ldimm64(&insns[i]))
+			i++;
+	}
+}
+
 int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	      struct bpf_log_attr *attr_log)
 {
@@ -23073,6 +23107,9 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (ret < 0)
 		goto skip_full_check;
 
+	if (env->log.level & BPF_LOG_LEVEL2)
+		log_program(env);
+
 	ret = mark_fastcall_patterns(env);
 	if (ret < 0)
 		goto skip_full_check;
diff --git a/tools/testing/selftests/bpf/progs/verifier_live_stack.c b/tools/testing/selftests/bpf/progs/verifier_live_stack.c
index 3abe08cc6595..6e8f50618580 100644
--- a/tools/testing/selftests/bpf/progs/verifier_live_stack.c
+++ b/tools/testing/selftests/bpf/progs/verifier_live_stack.c
@@ -66,7 +66,7 @@ __msg("stack use/def subprog#0 must_write_not_same_slot (d0,cs0):")
  * but the write conservatively marks the whole frame as may_def.
  */
 __msg("6: (7b) *(u64 *)(r2 +0) = r0         ; may_def: fp0-8..-{{(512|2048)}}")
-__msg("Live regs before insn:")
+__msg("Program dump")
 __naked void must_write_not_same_slot(void)
 {
 	asm volatile (

-- 
2.53.0

[PATCH bpf-next v3 23/51] bpf: compute immediate dominators

[Eduard Zingerman]

SCEV needs to find an exit condition that dominates a loop's backedge;
such conditions bound every loop iteration. Compute the immediate
dominator tree as a prerequisite for identifying such conditions.

Use the iterative reverse-postorder algorithm from:
"A Simple, Fast Dominance Algorithm" by Cooper et al.

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 include/linux/bpf_verifier.h |   8 +++
 kernel/bpf/Makefile          |   2 +-
 kernel/bpf/loops.c           | 158 +++++++++++++++++++++++++++++++++++++++++++
 kernel/bpf/verifier.c        |   8 ++-
 4 files changed, 174 insertions(+), 2 deletions(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index bd7733fbe930..59c0670af0e9 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -650,6 +650,11 @@ struct bpf_iarray {
 	u32 items[];
 };
 
+#define iarray_for_each(item, arr)						\
+	for (int ___idx = 0;							\
+	     ___idx < (arr)->cnt && ({ item = (arr)->items[___idx]; 1; });	\
+	     ___idx++)
+
 struct bpf_insn_aux_data {
 	union {
 		enum bpf_reg_type ptr_type;	/* pointer type for load/store insns */
@@ -1112,6 +1117,7 @@ struct bpf_verifier_env {
 	u32 scc_cnt;
 	struct bpf_iarray *succ;
 	struct bpf_iarray *gotox_tmp_buf;
+	int *idoms;
 };
 
 static inline struct bpf_func_info_aux *subprog_aux(struct bpf_verifier_env *env, int subprog)
@@ -1888,4 +1894,6 @@ int bpf_insn_def32(struct bpf_prog *prog, struct bpf_insn *insn);
 int bpf_flip_opcode(u32 opcode);
 u8 bpf_rev_opcode(u8 opcode);
 
+int bpf_compute_idoms(struct bpf_verifier_env *env);
+
 #endif /* _LINUX_BPF_VERIFIER_H */
diff --git a/kernel/bpf/Makefile b/kernel/bpf/Makefile
index c1f9b0d3468d..6210601eb40b 100644
--- a/kernel/bpf/Makefile
+++ b/kernel/bpf/Makefile
@@ -6,7 +6,7 @@ cflags-nogcse-$(CONFIG_X86)$(CONFIG_CC_IS_GCC) := -fno-gcse
 endif
 CFLAGS_core.o += -Wno-override-init $(cflags-nogcse-yy)
 
-obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o cnum.o log.o token.o liveness.o const_fold.o diagnostics.o
+obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o cnum.o log.o token.o liveness.o const_fold.o diagnostics.o loops.o
 obj-$(CONFIG_BPF_SYSCALL) += bpf_iter.o map_iter.o task_iter.o prog_iter.o link_iter.o
 obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o bloom_filter.o
 obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o bpf_insn_array.o
diff --git a/kernel/bpf/loops.c b/kernel/bpf/loops.c
new file mode 100644
index 000000000000..0db823ee42f7
--- /dev/null
+++ b/kernel/bpf/loops.c
@@ -0,0 +1,158 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+#include <linux/slab.h>
+#include <linux/sched/signal.h>
+#include <linux/bpf_verifier.h>
+
+static struct bpf_iarray **compute_predecessors(struct bpf_verifier_env *env)
+{
+	struct bpf_iarray *succ, *preds, **result;
+	struct bpf_prog *prog = env->prog;
+	u32 *num_preds, i, s, len = prog->len;
+	struct bpf_insn *insn;
+	size_t sz;
+	void *tmp;
+
+	num_preds = kvcalloc(prog->len, sizeof(u32), GFP_KERNEL_ACCOUNT);
+	if (!num_preds)
+		return NULL;
+
+	/*
+	 * 'result' layout:
+	 *  - array of pointers (struct bpf_iarray *)[len]
+	 *  - struct bpf_iarray one after another
+	 */
+	sz = size_mul(len, sizeof(*result) + sizeof(**result));
+	for (i = 0; i < len; i++) {
+		insn = env->prog->insnsi + i;
+		succ = bpf_insn_successors(env, i);
+		sz = size_add(sz, array_size(succ->cnt, sizeof(*succ->items)));
+		iarray_for_each(s, succ) {
+			num_preds[s]++;
+		}
+		if (bpf_is_ldimm64(insn))
+			i++;
+	}
+
+	result = kvzalloc(sz, GFP_KERNEL_ACCOUNT);
+	if (!result) {
+		kvfree(num_preds);
+		return NULL;
+	}
+
+	tmp = (void *)&result[len];
+	for (i = 0; i < len; i++) {
+		result[i] = tmp;
+		tmp += struct_size(result[i], items, num_preds[i]);
+	}
+
+	for (i = 0; i < len; i++) {
+		insn = env->prog->insnsi + i;
+		succ = bpf_insn_successors(env, i);
+		iarray_for_each(s, succ) {
+			preds = result[s];
+			preds->items[preds->cnt++] = i;
+		}
+		if (bpf_is_ldimm64(insn))
+			i++;
+	}
+
+	kvfree(num_preds);
+	return result;
+}
+
+static int idoms_intersect(struct bpf_verifier_env *env, int a, int b)
+{
+	int *postorder_nums = env->cfg.postorder_nums;
+	int *idoms = env->idoms;
+
+	while (a != b) {
+		while (postorder_nums[a] < postorder_nums[b]) {
+			a = idoms[a];
+		}
+		while (postorder_nums[b] < postorder_nums[a]) {
+			b = idoms[b];
+		}
+	}
+	return a;
+}
+
+/* See "A Simple, Fast Dominance Algorithm" by Cooper et al. for details. */
+static int compute_subprog_idoms(struct bpf_verifier_env *env, struct bpf_iarray **preds,
+				int subprog_idx)
+{
+	struct bpf_subprog_info *subprog = &env->subprog_info[subprog_idx];
+	int start = subprog->start;
+	int po_first = subprog->postorder_start;
+	int po_last = (subprog + 1)->postorder_start - 1;
+	int *idoms = env->idoms;
+	int po_num, pred;
+	u32 work = 0;
+	bool changed;
+
+	idoms[start] = start;
+	changed = true;
+	do {
+		changed = false;
+		/*
+		 * Iterate in reverse postorder, skipping the subprog entry:
+		 * insn_postorder[po_last] is the subprogram entry.
+		 */
+		for (po_num = po_last - 1; po_num >= po_first; po_num--) {
+			int idx = env->cfg.insn_postorder[po_num];
+			int new_idom = -1;
+
+			iarray_for_each(pred, preds[idx]) {
+				if (++work % 1024 == 0) {
+					if (signal_pending(current))
+						return -EAGAIN;
+					cond_resched();
+				}
+
+				if (idoms[pred] == -1)
+					continue;
+				if (new_idom == -1)
+					new_idom = pred;
+				else
+					new_idom = idoms_intersect(env, pred, new_idom);
+			}
+			if (new_idom != -1 && idoms[idx] != new_idom) {
+				idoms[idx] = new_idom;
+				changed = true;
+			}
+		}
+	} while (changed);
+	idoms[start] = -1;
+	return 0;
+}
+
+int bpf_compute_idoms(struct bpf_verifier_env *env)
+{
+	struct bpf_iarray **preds;
+	u32 len = env->prog->len;
+	int *idoms, i, err = 0;
+
+	preds = compute_predecessors(env);
+	if (!preds)
+		return -ENOMEM;
+
+	idoms = kvcalloc(len, sizeof(*idoms), GFP_KERNEL_ACCOUNT);
+	if (!idoms) {
+		kvfree(preds);
+		return -ENOMEM;
+	}
+
+	env->idoms = idoms;
+	for (i = 0; i < len; i++)
+		idoms[i] = -1;
+
+	for (i = 0; i < env->subprog_cnt; i++) {
+		err = compute_subprog_idoms(env, preds, i);
+		if (err)
+			break;
+	}
+
+	kvfree(preds);
+	return err;
+}
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 9260ee290072..5a29d05c523c 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -22872,13 +22872,14 @@ static void log_program(struct bpf_verifier_env *env)
 	u64 pos, insn_pos;
 	u32 i, j;
 
-	verbose(env, "Program dump (scc? insn#: live_regs_before):\n");
+	verbose(env, "Program dump (scc? idom insn#: live_regs_before):\n");
 	for (i = 0; i < insn_cnt; ++i) {
 		verbose_linfo(env, i, "    ; ");
 		if (env->insn_aux_data[i].scc)
 			verbose(env, "%3d ", env->insn_aux_data[i].scc);
 		else
 			verbose(env, "    ");
+		verbose(env, "%3d ", env->idoms[i]);
 		verbose(env, "%3d: ", i);
 		for (j = BPF_REG_0; j < BPF_REG_10; ++j)
 			if (insn_aux[i].live_regs_before & BIT(j))
@@ -23103,6 +23104,10 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 
 	prepare_subprog_args(env);
 
+	ret = bpf_compute_idoms(env);
+	if (ret < 0)
+		goto skip_full_check;
+
 	ret = bpf_compute_live_registers(env);
 	if (ret < 0)
 		goto skip_full_check;
@@ -23266,6 +23271,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	kvfree(env->callx_edges);
 	kvfree(env->func_ptrs);
 	bpf_diag_free(env);
+	kvfree(env->idoms);
 	kvfree(env);
 	return ret;
 }

-- 
2.53.0

[PATCH bpf-next v3 24/51] bpf: compute loop hierarchy

[Eduard Zingerman]

SCEV needs to process loops from innermost to outermost, summarizing
results for nested loops. Add a pass to compute the loop structure.

Use a non-recursive adaptation of the algorithm from:
"A New Algorithm for Identifying Loops in Decompilation" by Wei et al.

Record the loop hierarchy per instruction:
- The field insn_aux_data->loop_header records the innermost loop
  header containing it.
- The field insn_aux_data->loop records additional information about
  the loop if this instruction is a loop header (exits, backedges,
  irreducibility).

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 include/linux/bpf_verifier.h |  46 +++++
 kernel/bpf/fixups.c          |  18 ++
 kernel/bpf/loops.c           | 439 +++++++++++++++++++++++++++++++++++++++++++
 kernel/bpf/verifier.c        |  12 +-
 4 files changed, 514 insertions(+), 1 deletion(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 59c0670af0e9..d1ebbfef5455 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -655,6 +655,41 @@ struct bpf_iarray {
 	     ___idx < (arr)->cnt && ({ item = (arr)->items[___idx]; 1; });	\
 	     ___idx++)
 
+#define MAX_BACKEDGES 16
+#define MAX_LOOP_EXITS 256
+
+/* Number of exit conditions dominating a backedge. */
+#define MAX_LATCHES 4
+
+struct bpf_backedge {
+	int from;
+	/*
+	 * Conditional jumps within the loop that dominate 'from' and have
+	 * a successor outside the loop, nearest first. latches[0] is the
+	 * latch; the remaining entries are additional exit conditions.
+	 * Each usable condition gives an upper bound on the iteration count,
+	 * so keeping only a subset is sound.
+	 */
+	int latches[MAX_LATCHES];
+	u8 latches_cnt;
+};
+
+struct bpf_loop_exit {
+	int from; /* instruction inside the loop */
+	int to; /* instruction outside the loop */
+};
+
+struct bpf_loop {
+	struct bpf_backedge backedges[MAX_BACKEDGES];
+	/* edges exiting from this loop, includes edges from nested loops */
+	struct bpf_loop_exit *exits;
+	int backedges_cnt;
+	int exits_cnt;
+	bool irreducible;
+	bool backedges_overflow;
+	bool exits_overflow;
+};
+
 struct bpf_insn_aux_data {
 	union {
 		enum bpf_reg_type ptr_type;	/* pointer type for load/store insns */
@@ -752,6 +787,13 @@ struct bpf_insn_aux_data {
 	u16 const_reg_map_mask;
 	u16 const_reg_subprog_mask;
 	u32 const_reg_vals[10];
+	/*
+	 * Header index of the innermost loop containing this instruction, -1 if none.
+	 * For a loop header, identifies the parent loop instead.
+	 */
+	s32 loop_header;
+	/* additional information about the loop if this instruction is a loop header */
+	struct bpf_loop *loop;
 };
 
 #define MAX_USED_MAPS 64 /* max number of maps accessed by one eBPF program */
@@ -1879,6 +1921,7 @@ int bpf_check_attach_btf_id_multi(struct btf *btf, struct bpf_prog *prog, u32 bt
 				  struct bpf_attach_target_info *tgt_info);
 
 /* Functions in fixups.c, called from bpf_check() */
+void bpf_clear_insn_aux_data(struct bpf_verifier_env *env, int start, int len);
 int bpf_remove_fastcall_spills_fills(struct bpf_verifier_env *env);
 int bpf_optimize_bpf_loop(struct bpf_verifier_env *env);
 void bpf_opt_hard_wire_dead_code_branches(struct bpf_verifier_env *env);
@@ -1895,5 +1938,8 @@ int bpf_flip_opcode(u32 opcode);
 u8 bpf_rev_opcode(u8 opcode);
 
 int bpf_compute_idoms(struct bpf_verifier_env *env);
+int bpf_compute_loops(struct bpf_verifier_env *env);
+int bpf_loop_at_index(struct bpf_verifier_env *env, u32 idx);
+bool bpf_is_nested_loop(struct bpf_verifier_env *env, int inner_header, int outer_header);
 
 #endif /* _LINUX_BPF_VERIFIER_H */
diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
index 206cc9a614a9..b2fa7e8bedf3 100644
--- a/kernel/bpf/fixups.c
+++ b/kernel/bpf/fixups.c
@@ -590,6 +590,22 @@ static int bpf_adj_linfo_after_remove(struct bpf_verifier_env *env, u32 off,
 	return 0;
 }
 
+/* Clean up dynamically allocated fields of aux data for instructions [start, ...] */
+void bpf_clear_insn_aux_data(struct bpf_verifier_env *env, int start, int len)
+{
+	struct bpf_insn_aux_data *aux_data = env->insn_aux_data;
+	int end = start + len;
+	int i;
+
+	for (i = start; i < end; i++) {
+		if (aux_data[i].loop) {
+			kvfree(aux_data[i].loop->exits);
+			kvfree(aux_data[i].loop);
+			aux_data[i].loop = NULL;
+		}
+	}
+}
+
 static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt)
 {
 	struct bpf_insn_aux_data *aux_data = env->insn_aux_data;
@@ -602,6 +618,8 @@ static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt)
 	if (bpf_prog_is_offloaded(env->prog->aux))
 		bpf_prog_offload_remove_insns(env, off, cnt);
 
+	bpf_clear_insn_aux_data(env, off, cnt);
+
 	err = bpf_remove_insns(env->prog, off, cnt);
 	if (err)
 		return err;
diff --git a/kernel/bpf/loops.c b/kernel/bpf/loops.c
index 0db823ee42f7..8a6dc805e559 100644
--- a/kernel/bpf/loops.c
+++ b/kernel/bpf/loops.c
@@ -156,3 +156,442 @@ int bpf_compute_idoms(struct bpf_verifier_env *env)
 	kvfree(preds);
 	return err;
 }
+
+struct dfs_state {
+	u32 traversed:1;
+	u32 next_succ:31;
+};
+
+struct loops_dfs {
+	struct dfs_state *state;
+	int *dfs_pos;
+	int *stack;
+};
+
+static void mark_irreducible(struct bpf_verifier_env *env, int h)
+{
+	env->insn_aux_data[h].loop->irreducible = true;
+}
+
+static void add_backedge(struct bpf_verifier_env *env, int from, int h)
+{
+	struct bpf_insn_aux_data *aux = env->insn_aux_data;
+	struct bpf_loop *loop = aux[h].loop;
+	int cnt = loop->backedges_cnt;
+
+	if (cnt == MAX_BACKEDGES) {
+		loop->backedges_overflow = true;
+		return;
+	}
+	loop->backedges[cnt].from = from;
+	loop->backedges[cnt].latches_cnt = 0;
+	loop->backedges_cnt++;
+}
+
+static int add_exit(struct bpf_loop *loop, int from, int to)
+{
+	if (loop->exits_overflow)
+		return 0;
+	if (loop->exits_cnt == MAX_LOOP_EXITS) {
+		loop->exits_overflow = true;
+		return 0;
+	}
+	if (!loop->exits) {
+		loop->exits = kvcalloc(MAX_LOOP_EXITS, sizeof(*loop->exits), GFP_KERNEL_ACCOUNT);
+		if (!loop->exits)
+			return -ENOMEM;
+	}
+	loop->exits[loop->exits_cnt] = (struct bpf_loop_exit) {
+		.from = from,
+		.to = to,
+	};
+	loop->exits_cnt++;
+	return 0;
+}
+
+static int mark_as_header(struct bpf_verifier_env *env, int h)
+{
+	struct bpf_insn_aux_data *aux = env->insn_aux_data;
+
+	if (!aux[h].loop) {
+		aux[h].loop = kvzalloc_obj(struct bpf_loop, GFP_KERNEL_ACCOUNT);
+		if (!aux[h].loop)
+			return -ENOMEM;
+	}
+	return 0;
+}
+
+static int assign_header(struct bpf_verifier_env *env, struct loops_dfs *dfs, int n, int h)
+{
+	struct bpf_insn_aux_data *aux = env->insn_aux_data;
+	int *dfs_pos = dfs->dfs_pos;
+	int err, nh;
+
+	err = mark_as_header(env, h);
+	if (err)
+		return err;
+
+	/* Don't encode self-loops, otherwise can't reflect loops nesting structure. */
+	if (n == h)
+		return 0;
+
+	/* Make sure that loop headers up the chain are sorted by dfs_pos. */
+	while (aux[n].loop_header != -1) {
+		nh = aux[n].loop_header;
+		if (nh == h)
+			return 0;
+		if (dfs_pos[nh] < dfs_pos[h]) {
+			aux[n].loop_header = h;
+			n = h;
+			h = nh;
+		} else {
+			n = nh;
+		}
+	}
+	aux[n].loop_header = h;
+	return 0;
+}
+
+static bool is_cond_jmp_insn(struct bpf_insn *insn)
+{
+	u8 class = BPF_CLASS(insn->code);
+	u8 opcode = BPF_OP(insn->code);
+
+	if (class != BPF_JMP && class != BPF_JMP32)
+		return false;
+
+	switch (opcode) {
+	case BPF_JEQ:
+	case BPF_JGE:
+	case BPF_JGT:
+	case BPF_JLE:
+	case BPF_JLT:
+	case BPF_JNE:
+	case BPF_JSET:
+	case BPF_JSGE:
+	case BPF_JSGT:
+	case BPF_JSLE:
+	case BPF_JSLT:
+		return true;
+	default:
+		return false;
+	}
+}
+
+int bpf_loop_at_index(struct bpf_verifier_env *env, u32 idx)
+{
+	struct bpf_insn_aux_data *aux = env->insn_aux_data;
+
+	return aux[idx].loop ? idx : aux[idx].loop_header;
+}
+
+/*
+ * Collect conditional jumps within n's loop that dominate 'n' and have
+ * a successor outside the loop, nearest first.
+ */
+static void find_dominating_conditions(struct bpf_verifier_env *env, int n,
+				       struct bpf_backedge *backedge)
+{
+	struct bpf_insn *insns = env->prog->insnsi;
+	bool t_exits_loop, f_exits_loop;
+	int n_loop, t_loop, f_loop;
+	int *idoms = env->idoms;
+
+	n_loop = bpf_loop_at_index(env, n);
+	if (idoms_intersect(env, n, n_loop) != n_loop)
+		return;
+	while (n >= 0) {
+		if (is_cond_jmp_insn(&insns[n]) && bpf_loop_at_index(env, n) == n_loop) {
+			t_loop = bpf_loop_at_index(env, n + insns[n].off + 1);
+			f_loop = bpf_loop_at_index(env, n + 1);
+			t_exits_loop = t_loop != n_loop && !bpf_is_nested_loop(env, t_loop, n_loop);
+			f_exits_loop = f_loop != n_loop && !bpf_is_nested_loop(env, f_loop, n_loop);
+			if (t_exits_loop || f_exits_loop) {
+				if (backedge->latches_cnt == MAX_LATCHES)
+					return;
+				backedge->latches[backedge->latches_cnt++] = n;
+			}
+		}
+		if (n == n_loop)
+			break;
+		n = idoms[n];
+	}
+}
+
+/*
+ * As described in "A New Algorithm for Identifying Loops in Decompilation" by Wei et al,
+ * adapted to be non-recursive.
+ */
+static int compute_loops_in_subprog(struct bpf_verifier_env *env, struct loops_dfs *dfs,
+				    int subprog_idx)
+{
+	struct bpf_insn_aux_data *aux = env->insn_aux_data;
+	struct dfs_state *state = dfs->state;
+	int start = env->subprog_info[subprog_idx].start;
+	int *dfs_pos = dfs->dfs_pos;
+	int *stack = dfs->stack;
+	int s, h, err, cur, stack_sz;
+	struct bpf_iarray *succ;
+	u32 i;
+
+	stack[0] = start;
+	state[start].traversed = true;
+	state[start].next_succ = 0;
+	dfs_pos[start] = 1;
+	stack_sz = 1;
+	i = 0;
+	do {
+		/*
+		 * The algorithm should be very fast in practice,
+		 * guard against pathological inputs, just in case.
+		 */
+		if ((++i % 1024) == 0) {
+			if (signal_pending(current))
+				return -EAGAIN;
+			cond_resched();
+		}
+
+		cur = stack[stack_sz - 1];
+		succ = bpf_insn_successors(env, cur);
+		if (state[cur].next_succ == succ->cnt) {
+			dfs_pos[cur] = 0;
+			stack_sz--;
+			continue;
+		}
+		s = succ->items[state[cur].next_succ];
+		if (!state[s].traversed) {
+			/* Case A:  start -> ... -> cur -> s [unexplored] */
+			state[s].traversed = true;
+			state[s].next_succ = 0;
+			stack[stack_sz] = s;
+			dfs_pos[s] = stack_sz + 1;
+			stack_sz++;
+			continue;
+		}
+		/* 's' is fully explored at this point */
+		if (dfs_pos[s]) {
+			/*
+			 * start -> ... -> s -> cur --.
+			 *                 ^          |
+			 *                 '----------'
+			 * Case B: 's' is in the current DFS path.
+			 */
+			err = assign_header(env, dfs, cur, s);
+			if (err)
+				return err;
+			add_backedge(env, cur, s);
+		} else if (aux[s].loop_header == -1) {
+			/*
+			 * start -> ... -> ... -> s -> ... -> end
+			 *           |            ^
+			 *           '---> cur ---'
+			 * Case C: 's' is explored, not in the current DFS path,
+			 * and not a part of any loop.
+			 */
+		} else if (dfs_pos[aux[s].loop_header]) {
+			/*
+			 *                 .----------------------.
+			 *                 v                      |
+			 * start -> ... -> h -> ... -> ... -> s --'
+			 *                       |            ^
+			 *	                 '---> cur ---'
+			 * Case D: 's' is explored, not in current DFS path,
+			 * but its innermost loop header is.
+			 */
+			err = assign_header(env, dfs, cur, aux[s].loop_header);
+			if (err)
+				return err;
+		} else {
+			/*
+			 * case E: 's' is explored, not in current DFS path,
+			 * its innermost loop header is not in current DFS path,
+			 * hence 's' is another entry into the same loop.
+			 */
+			h = aux[s].loop_header;
+			mark_irreducible(env, h);
+			while (aux[h].loop_header != -1) {
+				h = aux[h].loop_header;
+				if (dfs_pos[h]) {
+					err = assign_header(env, dfs, cur, h);
+					if (err)
+						return err;
+					break;
+				}
+				mark_irreducible(env, h);
+			}
+		}
+		state[cur].next_succ++;
+	} while (stack_sz);
+
+	return 0;
+}
+
+bool bpf_is_nested_loop(struct bpf_verifier_env *env, int inner_header, int outer_header)
+{
+	struct bpf_insn_aux_data *aux = env->insn_aux_data;
+	int idx;
+
+	for (idx = inner_header; idx >= 0; idx = aux[idx].loop_header)
+		if (aux[idx].loop_header == outer_header)
+			return true;
+
+	return false;
+}
+
+int bpf_compute_loops(struct bpf_verifier_env *env)
+{
+	int i, j, k, s, t, iloop, sloop, err = 0, len = env->prog->len;
+	struct bpf_insn_aux_data *aux = env->insn_aux_data;
+	struct bpf_verifier_log *log = &env->log;
+	struct bpf_backedge *backedge;
+	struct loops_dfs dfs = {};
+	struct bpf_iarray *succ;
+	struct bpf_loop *loop;
+
+	dfs.dfs_pos = kvcalloc(len, sizeof(int), GFP_KERNEL_ACCOUNT);
+	dfs.state = kvcalloc(len, sizeof(struct dfs_state), GFP_KERNEL_ACCOUNT);
+	dfs.stack = kvcalloc(len, sizeof(int), GFP_KERNEL_ACCOUNT);
+	if (!dfs.dfs_pos || !dfs.state || !dfs.stack) {
+		err = -ENOMEM;
+		goto out;
+	}
+	for (i = 0; i < len; i++)
+		aux[i].loop_header = -1;
+	for (i = 0; i < env->subprog_cnt; i++) {
+		err = compute_loops_in_subprog(env, &dfs, i);
+		if (err)
+			goto out;
+	}
+	/* find latches */
+	for (i = 0; i < len; i++) {
+		loop = aux[i].loop;
+		if (!loop)
+			continue;
+		for (j = 0; j < loop->backedges_cnt; j++) {
+			backedge = &loop->backedges[j];
+			/*
+			 * In theory, the backedge->from and it's latch can reside in
+			 * an inner loop of `i`, e.g.:
+			 *
+			 *  1:  r7 += 1;
+			 *  2:  r6 += 1;
+			 *      if r7 == 2 goto 1b;
+			 *      if r6 < 2 goto 2b;
+			 *
+			 * For now, let's assume that latches are unknown for such cases.
+			 * (GCC/LLVM handle this by inserting artificial cfg nodes).
+			 */
+			if (bpf_loop_at_index(env, backedge->from) != i)
+				continue;
+			find_dominating_conditions(env, backedge->from, backedge);
+		}
+	}
+	/* find exits */
+	for (i = 0; i < len; i++) {
+		iloop = aux[i].loop ? i : aux[i].loop_header;
+		if (iloop < 0)
+			continue;
+		succ = bpf_insn_successors(env, i);
+		iarray_for_each(s, succ) {
+			/*
+			 * Nothing left to record once the innermost loop of 'i'
+			 * overflowed: the walk below would stop at it right away.
+			 */
+			if (aux[iloop].loop->exits_overflow)
+				break;
+			sloop = aux[s].loop ? s : aux[s].loop_header;
+			if (iloop == sloop)
+				continue;
+			if (bpf_is_nested_loop(env, sloop, iloop))
+				continue;
+			/*
+			 * At this point 'sloop' is either -1, an outer loop,
+			 * or a loop in another branch of the loop hierarchy.
+			 */
+			for (t = iloop; t >= 0 && t != sloop; t = aux[t].loop_header) {
+				/*
+				 * Account for the following configuration:
+				 *
+				 *   tloop {
+				 *     iloop {
+				 *       ... i: goto s;
+				 *     }
+				 *     sloop {
+				 *   s:
+				 *       ...
+				 *     }
+				 *   }
+				 */
+				if (bpf_is_nested_loop(env, sloop, t))
+					break;
+				/*
+				 * Record edges i -> s as exits from tloop when:
+				 *
+				 *   sloop {
+				 *     tloop {
+				 *       iloop {
+				 *         ... i: goto s;
+				 *       }
+				 *     }
+				 *  s: ...
+				 *   }
+				 */
+				/* Enclosing loops inherit the overflow, see below */
+				if (aux[t].loop->exits_overflow)
+					break;
+				err = add_exit(aux[t].loop, i, s);
+				if (err)
+					goto out;
+			}
+		}
+		if (bpf_is_ldimm64(env->prog->insnsi + i))
+			i++;
+	}
+	/* A nested loop with a truncated exit list can't be abstracted by SCEV */
+	for (i = 0; i < len; i++) {
+		loop = aux[i].loop;
+		if (!loop || !loop->exits_overflow)
+			continue;
+		for (t = aux[i].loop_header; t >= 0; t = aux[t].loop_header)
+			aux[t].loop->exits_overflow = true;
+	}
+
+	if (env->log.level & BPF_LOG_LEVEL2) {
+		for (i = 0; i < len; i++) {
+			loop = aux[i].loop;
+			if (!loop)
+				continue;
+			bpf_log(log, "loop at %d", i);
+			if (aux[i].loop_header >= 0)
+				bpf_log(log, ", nested in %d", aux[i].loop_header);
+			if (loop->irreducible)
+				bpf_log(log, ", irreducible");
+			if (loop->exits_overflow)
+				bpf_log(log, ", too many exits");
+			bpf_log(log, "\n");
+			for (j = 0; j < loop->backedges_cnt; j++) {
+				backedge = &loop->backedges[j];
+				bpf_log(log, "  backedge from %d", backedge->from);
+				for (k = 0; k < backedge->latches_cnt; k++) {
+					if (k == 0)
+						bpf_log(log, ", latch at ");
+					if (k == 1)
+						bpf_log(log, ", dominating exits at ");
+					if (k > 1)
+						bpf_log(log, ", ");
+					bpf_log(log, "%d", backedge->latches[k]);
+				}
+				bpf_log(log, "\n");
+			}
+			for (j = 0; j < loop->exits_cnt; j++)
+				bpf_log(log, "  exit from %d to %d\n",
+					loop->exits[j].from, loop->exits[j].to);
+		}
+	}
+
+out:
+	kvfree(dfs.dfs_pos);
+	kvfree(dfs.stack);
+	kvfree(dfs.state);
+	return err;
+}
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 5a29d05c523c..16b7758221b0 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -22872,13 +22872,17 @@ static void log_program(struct bpf_verifier_env *env)
 	u64 pos, insn_pos;
 	u32 i, j;
 
-	verbose(env, "Program dump (scc? idom insn#: live_regs_before):\n");
+	verbose(env, "Program dump (scc? loop_header? idom insn#: live_regs_before):\n");
 	for (i = 0; i < insn_cnt; ++i) {
 		verbose_linfo(env, i, "    ; ");
 		if (env->insn_aux_data[i].scc)
 			verbose(env, "%3d ", env->insn_aux_data[i].scc);
 		else
 			verbose(env, "    ");
+		if (env->insn_aux_data[i].loop_header >= 0)
+			verbose(env, "%3d ", env->insn_aux_data[i].loop_header);
+		else
+			verbose(env, "    ");
 		verbose(env, "%3d ", env->idoms[i]);
 		verbose(env, "%3d: ", i);
 		for (j = BPF_REG_0; j < BPF_REG_10; ++j)
@@ -23108,6 +23112,10 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (ret < 0)
 		goto skip_full_check;
 
+	ret = bpf_compute_loops(env);
+	if (ret < 0)
+		goto skip_full_check;
+
 	ret = bpf_compute_live_registers(env);
 	if (ret < 0)
 		goto skip_full_check;
@@ -23260,6 +23268,8 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	release_btfs(env);
 err_free_env:
 	bpf_free_subprog_jts(env);
+	if (env->insn_aux_data)
+		bpf_clear_insn_aux_data(env, 0, env->insn_aux_data_len);
 	vfree(env->insn_aux_data);
 	kvfree(env->fd_array);
 	bpf_stack_liveness_free(env);

-- 
2.53.0

Re: [PATCH bpf-next v3 24/51] bpf: compute loop hierarchy

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 6012 bytes --]

> bpf: compute loop hierarchy

TL;DR:
- Per-loop-header allocation (~2.4 KB) is unbounded, so a privileged 1M-insn
  program may use GiBs of memory before verification
- Latch and exit passes in bpf_compute_loops() lack
  signal_pending()/cond_resched(), so deeply nested loops may stall a load

> diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
> index 59c0670af0e9..d1ebbfef5455 100644
> --- a/include/linux/bpf_verifier.h
> +++ b/include/linux/bpf_verifier.h

[ ... ]

> @@ -655,6 +655,41 @@ struct bpf_iarray {
> +#define MAX_BACKEDGES 16
> +#define MAX_LOOP_EXITS 256

[ ... ]

> +struct bpf_loop {
> +	struct bpf_backedge backedges[MAX_BACKEDGES];
> +	/* edges exiting from this loop, includes edges from nested loops */
> +	struct bpf_loop_exit *exits;
> +	int backedges_cnt;
> +	int exits_cnt;
> +	bool irreducible;
> +	bool backedges_overflow;
> +	bool exits_overflow;
> +};

Each loop header gets its own struct bpf_loop with backedges[MAX_BACKEDGES]
embedded, which is roughly 408 bytes.  On the first exit, add_exit() also
allocates the full MAX_LOOP_EXITS array, 256 * sizeof(struct bpf_loop_exit) =
2048 bytes.  That is about 2.5 kilobytes per loop header, and there is no
limit on the number of loops in a program.

A privileged loader can submit up to BPF_COMPLEXITY_LIMIT_INSNS (1M)
instructions, and back-edges are accepted for it.  Consider 1M instructions
of the form:

    if r1 == 0 goto -1

followed by exit.  Every instruction is a self-loop header, so
compute_loops_in_subprog() calls assign_header() and then mark_as_header()
for each one, which does the kvzalloc_obj().  Each one also has an exit edge
to the next instruction, so bpf_compute_loops() calls add_exit(), which does
the 2048 byte kvcalloc().

Can a single BPF_PROG_LOAD allocate about 2.4 GiB (around 2 million slab
objects) this way before do_check() gets a chance to reject the program?

The existing per-instruction struct bpf_insn_aux_data is about 160 bytes, so
this is more than ten times the verifier's per-instruction footprint.  The
allocations use GFP_KERNEL_ACCOUNT, but privileged loads are not serialized
by bpf_verifier_lock, so concurrent loaders add up.  The commit message does
not mention the memory cost.

Would it make sense to size the exits array on demand, and to keep the
rarely used backedges array out of line?

[ ... ]

> diff --git a/kernel/bpf/loops.c b/kernel/bpf/loops.c
> index 0db823ee42f7..8a6dc805e559 100644
> --- a/kernel/bpf/loops.c
> +++ b/kernel/bpf/loops.c

[ ... ]

> +static int add_exit(struct bpf_loop *loop, int from, int to)
> +{
> +	if (loop->exits_overflow)
> +		return 0;
> +	if (loop->exits_cnt == MAX_LOOP_EXITS) {
> +		loop->exits_overflow = true;
> +		return 0;
> +	}
> +	if (!loop->exits) {
> +		loop->exits = kvcalloc(MAX_LOOP_EXITS, sizeof(*loop->exits), GFP_KERNEL_ACCOUNT);
> +		if (!loop->exits)
> +			return -ENOMEM;
> +	}
> +	loop->exits[loop->exits_cnt] = (struct bpf_loop_exit) {
> +		.from = from,
> +		.to = to,
> +	};
> +	loop->exits_cnt++;
> +	return 0;
> +}

Related to the above, add_exit() allocates the full MAX_LOOP_EXITS array
(256 entries, 2048 bytes) on the first exit of a loop, even though most loops
have only one or two exits.

Together with the struct bpf_loop allocated in mark_as_header(), this is
about 2.4 kilobytes per loop header, all of it charged with
GFP_KERNEL_ACCOUNT and alive until the end of bpf_check().

Could the exits array start small and double on demand up to MAX_LOOP_EXITS?
That would keep the same overflow semantics.

[ ... ]

> +int bpf_compute_loops(struct bpf_verifier_env *env)

[ ... ]

> +	/* find latches */
> +	for (i = 0; i < len; i++) {
> +		loop = aux[i].loop;
> +		if (!loop)
> +			continue;
> +		for (j = 0; j < loop->backedges_cnt; j++) {

[ ... ]

> +			find_dominating_conditions(env, backedge->from, backedge);
> +		}
> +	}
> +	/* find exits */
> +	for (i = 0; i < len; i++) {
> +		iloop = aux[i].loop ? i : aux[i].loop_header;
> +		if (iloop < 0)
> +			continue;
> +		succ = bpf_insn_successors(env, i);
> +		iarray_for_each(s, succ) {

[ ... ]

> +			if (bpf_is_nested_loop(env, sloop, iloop))
> +				continue;

[ ... ]

> +			for (t = iloop; t >= 0 && t != sloop; t = aux[t].loop_header) {

[ ... ]

> +				if (bpf_is_nested_loop(env, sloop, t))
> +					break;

Do the "find latches" and "find exits" passes need the same
signal_pending()/cond_resched() guard that compute_loops_in_subprog() has?

compute_loops_in_subprog() does this:

    if ((++i % 1024) == 0) {
            if (signal_pending(current))
                    return -EAGAIN;
            cond_resched();
    }

with the comment "guard against pathological inputs, just in case", and
compute_subprog_idoms() in the preceding commit does the same.  The two outer
loops over 'len' in bpf_compute_loops() have no such point.

For each exit edge, bpf_is_nested_loop() walks the loop_header chain of
sloop, and the 'for (t = iloop; ...)' walk calls it again for every
enclosing loop, so the cost is O(depth) to O(depth^2) per edge.  Likewise,
find_dominating_conditions() calls idoms_intersect() and walks idoms[] from
each backedge source up to its header.

Privileged programs may have up to BPF_COMPLEXITY_LIMIT_INSNS (1M)
instructions and back-edges are allowed for them, so these walks are bounded
only by program size.  This pass also runs before do_check(), so the
preemption and signal checks in do_check() do not apply.

On a non-preemptible kernel, can a deeply nested input make a single
BPF_PROG_LOAD run for a long time without rescheduling and without being
interruptible by a signal?

Would a periodic signal_pending()/cond_resched() check returning -EAGAIN in
the two outer 'for (i = 0; i < len; i++)' loops be appropriate, to match the
sibling passes?


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/37923187511

[PATCH bpf-next v3 25/51] bpf: add bpf_set_reg_range()

[Eduard Zingerman]

A utility function for setting a register's range and step information.
Used by SCEV widening and clamping logic further in the series.

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 include/linux/bpf_verifier.h |  2 ++
 kernel/bpf/verifier.c        | 12 ++++++++++++
 2 files changed, 14 insertions(+)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index d1ebbfef5455..bc901d649472 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1820,6 +1820,8 @@ struct arg_access_info
 bpf_global_subprog_stack_access_bytes(struct bpf_verifier_env *env,
 				      struct bpf_insn *insn, int arg, int insn_idx);
 int bpf_compute_subprog_arg_access(struct bpf_verifier_env *env);
+int bpf_set_reg_range(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
+		      struct cnum64 range, u16 base, u16 step);
 
 int bpf_stack_liveness_init(struct bpf_verifier_env *env);
 void bpf_stack_liveness_free(struct bpf_verifier_env *env);
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 16b7758221b0..585303d28690 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -17273,6 +17273,18 @@ static int adjust_reg_min_max_vals(struct bpf_verifier_env *env,
 	return 0;
 }
 
+int bpf_set_reg_range(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
+		      struct cnum64 range, u16 base, u16 step)
+{
+	reg->r64 = range;
+	reg->r32 = CNUM32_UNBOUNDED;
+	reg->step = step;
+	reg->base = base;
+	reg->var_off = tnum_unknown;
+	reg_bounds_sync(reg); /* this should infer the tnum alignment */
+	return reg_bounds_sanity_check(env, reg, "bpf_set_reg_range");
+}
+
 /* check validity of 32-bit and 64-bit arithmetic operations */
 static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn)
 {

-- 
2.53.0

[PATCH bpf-next v3 26/51] bpf: add bpf_mark_reg_known_scalar()

[Eduard Zingerman]

Provide a helper to set a register to a known scalar value for use outside
verifier.c. To be used by SCEV widening logic.

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 include/linux/bpf_verifier.h | 1 +
 kernel/bpf/verifier.c        | 6 ++++++
 2 files changed, 7 insertions(+)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index bc901d649472..c7b7c8148205 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1353,6 +1353,7 @@ int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state
 void bpf_bt_sync_linked_regs(struct backtrack_state *bt, struct bpf_jmp_history_entry *hist);
 void bpf_mark_reg_not_init(const struct bpf_verifier_env *env,
 			   struct bpf_reg_state *reg);
+void bpf_mark_reg_known_scalar(struct bpf_reg_state *reg, u64 imm);
 void bpf_mark_reg_unknown_imprecise(struct bpf_reg_state *reg);
 int bpf_grow_stack_state(struct bpf_verifier_env *env, struct bpf_func_state *state, int size);
 void bpf_mark_all_scalars_precise(struct bpf_verifier_env *env,
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 585303d28690..743fc2f3c24b 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -2057,6 +2057,12 @@ static void __mark_reg_known(struct bpf_reg_state *reg, u64 imm)
 	___mark_reg_known(reg, imm);
 }
 
+void bpf_mark_reg_known_scalar(struct bpf_reg_state *reg, u64 imm)
+{
+	__mark_reg_known(reg, imm);
+	reg->type = SCALAR_VALUE;
+}
+
 static void __mark_reg32_known(struct bpf_reg_state *reg, u64 imm)
 {
 	reg->var_off = tnum_const_subreg(reg->var_off, imm);

-- 
2.53.0

[PATCH bpf-next v3 27/51] bpf: add bpf_reg_union() and bpf_reg_union_compatible()

[Eduard Zingerman]

Utility functions to take a union of two registers' scalar values:
merge the circular 32-bit and 64-bit bounds, tnums and base/step
of two registers, then synchronize and validate the result.
Leave id invalidation and type validation to the caller.
Caller should use bpf_reg_union_compatible() to ensure
that final union result is sensible.

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 include/linux/bpf_verifier.h |  3 +++
 kernel/bpf/verifier.c        | 46 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 49 insertions(+)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index c7b7c8148205..1d0a0b64b1f3 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1823,6 +1823,9 @@ bpf_global_subprog_stack_access_bytes(struct bpf_verifier_env *env,
 int bpf_compute_subprog_arg_access(struct bpf_verifier_env *env);
 int bpf_set_reg_range(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
 		      struct cnum64 range, u16 base, u16 step);
+bool bpf_reg_union_compatible(const struct bpf_reg_state *a, const struct bpf_reg_state *b);
+int bpf_reg_union(struct bpf_verifier_env *env, struct bpf_reg_state *acc,
+		  const struct bpf_reg_state *src);
 
 int bpf_stack_liveness_init(struct bpf_verifier_env *env);
 void bpf_stack_liveness_free(struct bpf_verifier_env *env);
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 743fc2f3c24b..f8e6b40c2121 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -31,6 +31,7 @@
 #include <linux/module.h>
 #include <linux/cpumask.h>
 #include <linux/cnum.h>
+#include <linux/gcd.h>
 #include <linux/bpf_mem_alloc.h>
 #include <net/xdp.h>
 #include <linux/trace_events.h>
@@ -17291,6 +17292,51 @@ int bpf_set_reg_range(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
 	return reg_bounds_sanity_check(env, reg, "bpf_set_reg_range");
 }
 
+/*
+ * Checks if pointer and packet metadata matches between two registers.
+ * For use with bpf_reg_union().
+ */
+bool bpf_reg_union_compatible(const struct bpf_reg_state *a, const struct bpf_reg_state *b)
+{
+	if (a->type != b->type)
+		return false;
+	if (a->type == SCALAR_VALUE)
+		return true;
+	return !memcmp(a, b, offsetof(struct bpf_reg_state, var_off)) &&
+	       a->id == b->id && a->parent_id == b->parent_id && a->map_uid == b->map_uid;
+}
+
+/*
+ * Extends acc's scalar range to include src's scalar range.
+ * Caller must ensure that other kinds of metadata is compatible.
+ * Caller must clear acc's scalar ID.
+ */
+int bpf_reg_union(struct bpf_verifier_env *env, struct bpf_reg_state *acc,
+		  const struct bpf_reg_state *src)
+{
+	u16 base, step;
+
+	if (!bpf_reg_union_compatible(acc, src)) {
+		verifier_bug(env, "union of incompatible registers");
+		return -EFAULT;
+	}
+	acc->r64 = cnum64_union(acc->r64, src->r64);
+	acc->r32 = cnum32_union(acc->r32, src->r32);
+	acc->var_off = tnum_union(acc->var_off, src->var_off);
+
+	/* Retain a common congruence if the bases agree modulo the gcd. */
+	step = gcd(acc->step, src->step);
+	base = acc->base % step;
+	if (base != src->base % step) {
+		reg_step_reset(acc);
+	} else {
+		acc->base = base;
+		acc->step = step;
+	}
+	reg_bounds_sync(acc);
+	return reg_bounds_sanity_check(env, acc, "bpf_reg_union");
+}
+
 /* check validity of 32-bit and 64-bit arithmetic operations */
 static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn)
 {

-- 
2.53.0

[PATCH bpf-next v3 28/51] bpf: add bpf_split_cur_state()

[Eduard Zingerman]

SCEV widening needs an explicit checkpoint preserving the original
loop-entry state, independently of ordinary state-cache heuristics.
This commit extracts checkpoint creation logic from
bpf_is_state_visited() into a separate utility function.

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 include/linux/bpf_verifier.h |  1 +
 kernel/bpf/states.c          | 95 ++++++++++++++++++++++++--------------------
 2 files changed, 54 insertions(+), 42 deletions(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 1d0a0b64b1f3..711a4cb24c06 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1339,6 +1339,7 @@ void bpf_free_kfunc_btf_tab(struct bpf_kfunc_btf_tab *tab);
 int mark_chain_precision(struct bpf_verifier_env *env, int regno);
 
 int bpf_is_state_visited(struct bpf_verifier_env *env, int insn_idx);
+int bpf_split_cur_state(struct bpf_verifier_env *env);
 int bpf_update_branch_counts(struct bpf_verifier_env *env, struct bpf_verifier_state *st);
 
 void bpf_clear_jmp_history(struct bpf_verifier_state *state);
diff --git a/kernel/bpf/states.c b/kernel/bpf/states.c
index 2f6a7620eb72..b1e2daefc264 100644
--- a/kernel/bpf/states.c
+++ b/kernel/bpf/states.c
@@ -1262,11 +1262,61 @@ static void mark_all_scalars_imprecise(struct bpf_verifier_env *env, struct bpf_
 	}
 }
 
-int bpf_is_state_visited(struct bpf_verifier_env *env, int insn_idx)
+int bpf_split_cur_state(struct bpf_verifier_env *env)
 {
+	struct bpf_verifier_state *cur = env->cur_state, *new;
 	struct bpf_verifier_state_list *new_sl;
+	struct list_head *head;
+	int insn_idx = cur->insn_idx;
+	int err;
+
+	head = bpf_explored_state(env, insn_idx);
+	new_sl = kzalloc_obj(struct bpf_verifier_state_list, GFP_KERNEL_ACCOUNT);
+	if (!new_sl)
+		return -ENOMEM;
+	env->total_states++;
+	env->explored_states_size++;
+	update_peak_states(env);
+	env->prev_jmps_processed = env->jmps_processed;
+	env->prev_insn_processed = env->insn_processed;
+
+	/* forget precise markings we inherited, see __mark_chain_precision */
+	if (env->bpf_capable)
+		mark_all_scalars_imprecise(env, cur);
+
+	bpf_clear_singular_ids(env, cur);
+
+	/* add new state to the head of linked list */
+	new = &new_sl->state;
+	err = bpf_copy_verifier_state(new, cur);
+	if (err) {
+		bpf_free_verifier_state(new, false);
+		kfree(new_sl);
+		return err;
+	}
+	new->insn_idx = insn_idx;
+	verifier_bug_if(new->branches != 1, env,
+			"%s:branches_to_explore=%d insn %d",
+			__func__, new->branches, insn_idx);
+	err = maybe_enter_scc(env, new);
+	if (err) {
+		bpf_free_verifier_state(new, false);
+		kfree(new_sl);
+		return err;
+	}
+
+	cur->parent = new;
+	cur->first_insn_idx = insn_idx;
+	cur->dfs_depth = new->dfs_depth + 1;
+	bpf_clear_jmp_history(cur);
+	list_add(&new_sl->node, head);
+	return 0;
+}
+
+int bpf_is_state_visited(struct bpf_verifier_env *env, int insn_idx)
+{
 	struct bpf_verifier_state_list *sl;
-	struct bpf_verifier_state *cur = env->cur_state, *new;
+	struct bpf_verifier_state *cur = env->cur_state;
 	bool force_new_state, add_new_state, loop;
 	int n, err, states_cnt = 0;
 	struct list_head *pos, *tmp, *head;
@@ -1594,44 +1644,5 @@ int bpf_is_state_visited(struct bpf_verifier_env *env, int insn_idx)
 	 * When looping the sl->state.branches will be > 0 and this state
 	 * will not be considered for equivalence until branches == 0.
 	 */
-	new_sl = kzalloc_obj(struct bpf_verifier_state_list, GFP_KERNEL_ACCOUNT);
-	if (!new_sl)
-		return -ENOMEM;
-	env->total_states++;
-	env->explored_states_size++;
-	update_peak_states(env);
-	env->prev_jmps_processed = env->jmps_processed;
-	env->prev_insn_processed = env->insn_processed;
-
-	/* forget precise markings we inherited, see __mark_chain_precision */
-	if (env->bpf_capable)
-		mark_all_scalars_imprecise(env, cur);
-
-	bpf_clear_singular_ids(env, cur);
-
-	/* add new state to the head of linked list */
-	new = &new_sl->state;
-	err = bpf_copy_verifier_state(new, cur);
-	if (err) {
-		bpf_free_verifier_state(new, false);
-		kfree(new_sl);
-		return err;
-	}
-	new->insn_idx = insn_idx;
-	verifier_bug_if(new->branches != 1, env,
-			"%s:branches_to_explore=%d insn %d",
-			__func__, new->branches, insn_idx);
-	err = maybe_enter_scc(env, new);
-	if (err) {
-		bpf_free_verifier_state(new, false);
-		kfree(new_sl);
-		return err;
-	}
-
-	cur->parent = new;
-	cur->first_insn_idx = insn_idx;
-	cur->dfs_depth = new->dfs_depth + 1;
-	bpf_clear_jmp_history(cur);
-	list_add(&new_sl->node, head);
-	return 0;
+	return bpf_split_cur_state(env);
 }

-- 
2.53.0

[PATCH bpf-next v3 29/51] bpf: add bpf_same_memory_origin()

[Eduard Zingerman]

When dealing with loops like:

  for (ptr = ...; ptr < end_ptr; ptr += 8)
    ...

Scalar evolution needs to know if ptr and end_ptr identify a same
memory location (possibly with different offsets). If they do,
it is sound to attempt to infer iteration bounds for such a loop.

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 include/linux/bpf_verifier.h |  2 ++
 kernel/bpf/verifier.c        | 49 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 51 insertions(+)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 711a4cb24c06..6daa0c826f25 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1822,6 +1822,8 @@ struct arg_access_info
 bpf_global_subprog_stack_access_bytes(struct bpf_verifier_env *env,
 				      struct bpf_insn *insn, int arg, int insn_idx);
 int bpf_compute_subprog_arg_access(struct bpf_verifier_env *env);
+bool bpf_same_memory_origin(const struct bpf_reg_state *reg_a,
+			    const struct bpf_reg_state *reg_b);
 int bpf_set_reg_range(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
 		      struct cnum64 range, u16 base, u16 step);
 bool bpf_reg_union_compatible(const struct bpf_reg_state *a, const struct bpf_reg_state *b);
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index f8e6b40c2121..bf0722752e76 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -17280,6 +17280,55 @@ static int adjust_reg_min_max_vals(struct bpf_verifier_env *env,
 	return 0;
 }
 
+/*
+ * Many checks done by this function are quite conservative.
+ * This is because main verification pass does not maintain
+ * enough information to track object identities for some
+ * of the interesting types, e.g. PTR_TO_MEM.
+ */
+bool bpf_same_memory_origin(const struct bpf_reg_state *reg_a,
+			    const struct bpf_reg_state *reg_b)
+{
+	if (reg_a == reg_b)
+		return true;
+	/* Require matching flags and base types. */
+	if (reg_a->type != reg_b->type)
+		return false;
+	/* NULL can't be compared to some base+offset pointer. */
+	if (type_may_be_null(reg_a->type))
+		return false;
+
+	switch (base_type(reg_a->type)) {
+	case PTR_TO_STACK:
+		return reg_a->frameno == reg_b->frameno;
+	case PTR_TO_MAP_VALUE:
+		if (reg_a->map_ptr != reg_b->map_ptr || reg_a->map_uid != reg_b->map_uid)
+			return false;
+		if (reg_a->id && reg_a->id == reg_b->id)
+			return true;
+		/*
+		 * A plain single-element array has one stable value address,
+		 * but not when BPF_F_INNER_MAP is set, see arraymap.c:array_map_meta_equal().
+		 */
+		if (reg_a->map_ptr->map_type == BPF_MAP_TYPE_ARRAY &&
+		    reg_a->map_ptr->max_entries == 1 &&
+		    !(reg_a->map_ptr->map_flags & BPF_F_INNER_MAP))
+			return true;
+		/*
+		 * The rules above can be simplified / relaxed if:
+		 * - fresh IDs would always be assigned for map-value lookups;
+		 * - direct map value loads would always have and ID of zero.
+		 */
+		return false;
+	case PTR_TO_MEM:
+	case PTR_TO_BUF:
+		return reg_a->id && reg_a->id == reg_b->id;
+
+	default:
+		return false;
+	}
+}
+
 int bpf_set_reg_range(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
 		      struct cnum64 range, u16 base, u16 step)
 {

-- 
2.53.0

[PATCH bpf-next v3 30/51] bpf: add bpf_spill_size()

[Eduard Zingerman]

A utility function counting the number of STACK_SPILL slots in a
bpf_stat_state instance. For use by SCEV widening logic.

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 include/linux/bpf_verifier.h | 9 +++++++++
 kernel/bpf/verifier.c        | 5 +----
 2 files changed, 10 insertions(+), 4 deletions(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 6daa0c826f25..7fb1e315de48 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1420,6 +1420,15 @@ static inline bool bpf_is_spilled_reg(const struct bpf_stack_state *stack)
 	return stack->slot_type[BPF_REG_SIZE - 1] == STACK_SPILL;
 }
 
+static inline int bpf_spill_size(const struct bpf_stack_state *stack)
+{
+	int i, spill_size = 0;
+
+	for (i = BPF_REG_SIZE - 1; i >= 0 && stack->slot_type[i] == STACK_SPILL; i--)
+		spill_size++;
+	return spill_size;
+}
+
 static inline bool bpf_is_spilled_scalar_reg(const struct bpf_stack_state *stack)
 {
 	return bpf_is_spilled_reg(stack) && stack->spilled_ptr.type == SCALAR_VALUE;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index bf0722752e76..85971f9c09c4 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -4247,10 +4247,7 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env,
 		bpf_diag_mod_begin(env, &state->regs[dst_regno], reg, BPF_DIAG_MOD_WRITE);
 
 	if (bpf_is_spilled_reg(ss)) {
-		u8 spill_size = 1;
-
-		for (i = BPF_REG_SIZE - 1; i > 0 && stype[i - 1] == STACK_SPILL; i--)
-			spill_size++;
+		u8 spill_size = bpf_spill_size(ss);
 
 		if (size != BPF_REG_SIZE || spill_size != BPF_REG_SIZE) {
 			if (reg->type != SCALAR_VALUE) {

-- 
2.53.0

[PATCH bpf-next v3 31/51] bpf: record basic-block ends in insn_aux_data

[Eduard Zingerman]

From: Eduard Zingerman <ezingerman@fb.com>

SCEV needs information about the program's basic-block structure.
Piggyback on compute_predecessors() and set
insn_aux_data->bb_end for an instruction if:
- it has multiple successors, or
- its sole successor has multiple predecessors, or
- it is a jump instruction.

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 include/linux/bpf_verifier.h |  1 +
 kernel/bpf/loops.c           | 19 +++++++++++++++++++
 2 files changed, 20 insertions(+)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 7fb1e315de48..1458e50a1ad1 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -741,6 +741,7 @@ struct bpf_insn_aux_data {
 	bool is_iter_next; /* bpf_iter_<type>_next() kfunc call */
 	bool call_with_percpu_alloc_ptr; /* {this,per}_cpu_ptr() with prog percpu alloc */
 	bool arena_scalar; /* ldx/stx/st/atomic through a number, it's an address in arena */
+	bool bb_end; /* last instruction of a basic block */
 	u8 alu_state; /* used in combination with alu_limit */
 	/* true if STX or LDX instruction is a part of a spill/fill
 	 * pattern for a bpf_fastcall call.
diff --git a/kernel/bpf/loops.c b/kernel/bpf/loops.c
index 8a6dc805e559..2568079c2945 100644
--- a/kernel/bpf/loops.c
+++ b/kernel/bpf/loops.c
@@ -5,8 +5,23 @@
 #include <linux/sched/signal.h>
 #include <linux/bpf_verifier.h>
 
+static bool is_cfg_jump(struct bpf_insn *insn)
+{
+	u8 class = BPF_CLASS(insn->code);
+	u8 opcode = BPF_OP(insn->code);
+
+	switch (class) {
+	case BPF_JMP:
+	case BPF_JMP32:
+		return opcode != BPF_CALL;
+	default:
+		return false;
+	}
+}
+
 static struct bpf_iarray **compute_predecessors(struct bpf_verifier_env *env)
 {
+	struct bpf_insn_aux_data *aux = env->insn_aux_data;
 	struct bpf_iarray *succ, *preds, **result;
 	struct bpf_prog *prog = env->prog;
 	u32 *num_preds, i, s, len = prog->len;
@@ -54,6 +69,10 @@ static struct bpf_iarray **compute_predecessors(struct bpf_verifier_env *env)
 			preds = result[s];
 			preds->items[preds->cnt++] = i;
 		}
+		if (succ->cnt > 1 ||
+		    (succ->cnt == 1 && num_preds[succ->items[0]] > 1) ||
+		    is_cfg_jump(insn))
+			aux[i].bb_end = true;
 		if (bpf_is_ldimm64(insn))
 			i++;
 	}

-- 
2.53.0

Re: [PATCH bpf-next v3 31/51] bpf: record basic-block ends in insn_aux_data

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 1275 bytes --]

> bpf: record basic-block ends in insn_aux_data

TL;DR: Commit message only: author email (ezingerman@fb.com) doesn't match the
Signed-off-by (eddyz87@gmail.com); no code bug. Reset the author or add a
matching Signed-off-by?

> bpf: record basic-block ends in insn_aux_data
>
> SCEV needs information about the program's basic-block structure.
> Piggyback on compute_predecessors() and set
> insn_aux_data->bb_end for an instruction if:
> - it has multiple successors, or
> - its sole successor has multiple predecessors, or
> - it is a jump instruction.
>
> Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>

This isn't a bug in the code, but the commit author is
Eduard Zingerman <ezingerman@fb.com> while the only Signed-off-by is
Eduard Zingerman <eddyz87@gmail.com>.

checkpatch.pl reports this as a "From:/Signed-off-by: email address
mismatch".  The other commits in this series use eddyz87@gmail.com as the
author.

Should the author be reset to eddyz87@gmail.com, or should a Signed-off-by
for ezingerman@fb.com be added?


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/37923187511

[PATCH bpf-next v3 32/51] bpf: allow precision backtracking between overlapping checkpoints

[Eduard Zingerman]

SCEV integration logic might create two consecutive checkpoints at the
same instruction w/o processing any instructions in between.

This happens for loop-entry checkpoint -> regular checkpoint
sequences, where the loop-entry checkpoint has to remain unchanged
but the regular checkpoint is used for widening.

This commit adapts bpf_mark_chain_precision() to support this.

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 include/linux/bpf_verifier.h | 5 ++++-
 kernel/bpf/backtrack.c       | 5 +++++
 kernel/bpf/states.c          | 1 +
 3 files changed, 10 insertions(+), 1 deletion(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 1458e50a1ad1..963106328791 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -502,7 +502,10 @@ struct bpf_verifier_state {
 	bool speculative;
 	bool in_sleepable;
 
-	/* first and last insn idx of this verifier state */
+	/*
+	 * First and last insn idx of this verifier state.
+	 * last_insn_idx is -1 if no instructions have been executed yet.
+	 */
 	u32 first_insn_idx;
 	u32 last_insn_idx;
 	/* if this state is a backedge state then equal_state
diff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c
index 0e38b9575328..43d5740034c0 100644
--- a/kernel/bpf/backtrack.c
+++ b/kernel/bpf/backtrack.c
@@ -888,6 +888,10 @@ int bpf_mark_chain_precision(struct bpf_verifier_env *env,
 		}
 
 		if (last_idx < 0) {
+			/* Consecutive checkpoints can have no instructions between them. */
+			if (st->parent)
+				goto parent;
+
 			/* we are at the entry into subprog, which
 			 * is expected for global funcs, but only if
 			 * requested precise registers are R1-R5
@@ -952,6 +956,7 @@ int bpf_mark_chain_precision(struct bpf_verifier_env *env,
 				return -EFAULT;
 			}
 		}
+parent:
 		st = st->parent;
 		if (!st)
 			break;
diff --git a/kernel/bpf/states.c b/kernel/bpf/states.c
index b1e2daefc264..470bff8d8dc8 100644
--- a/kernel/bpf/states.c
+++ b/kernel/bpf/states.c
@@ -1306,6 +1306,7 @@ int bpf_split_cur_state(struct bpf_verifier_env *env)
 	}
 
 	cur->parent = new;
+	cur->last_insn_idx = -1;
 	cur->first_insn_idx = insn_idx;
 	cur->dfs_depth = new->dfs_depth + 1;
 	bpf_clear_jmp_history(cur);

-- 
2.53.0

[PATCH bpf-next v3 33/51] bpf: compute scalar evolution expressions for loops

[Eduard Zingerman]

Assign algebraic expressions to loop variables (registers and stack
spills), describing how their values evolve across iterations. These
summaries provide symbolic input for the subsequent widening patch.

The algorithm draws on ideas from the following papers:

- "Symbolic Evaluation of Chains of Recurrences for Loop Optimization"
  Robert A. van Engelen, 2000
- "The CR# Algebra and its Application in Loop Analysis and Optimization"
  Robert A. van Engelen, 2004

The analysis proceeds in two phases:

- Compute expressions for one symbolic iteration: start each variable
  with a reference to its input value, apply instruction effects and
  join expressions where control-flow paths merge.
- Convert the resulting backedge updates to recurrences at the header,
  then substitute these recurrences into expressions at the latch.

Analyze loops innermost first. Within each loop, visit blocks in
topological order (reverse postorder, with backedges ignored and nested
loops collapsed). Treat each nested loop as an opaque operation:
preserve its invariant values, forget the others and continue at its
exits, without expanding its iterations.

For example, using 64-bit arithmetic:

    0: r7 = 5;
    1: r6 = 0;
    2: do {                // header
    3:     if (r6 == 1)
    4:         r7 = 10;
    5:     r6 += 1;
    6: } while (r6 < 3);   // latch

Expressions use symbolic inputs r6 and r7; the concrete values from
lines 0-1 are supplied later by the main verifier.

First phase:
------------

At each instruction, an environment maps register numbers to their
current expressions. Transfer functions update this mapping, while
joins merge the environments arriving along different paths.

For illustration, follow each branch separately to the backedge:

- At 3, take the path skipping 4: r6 = r6, r7 = r7.
- At 5, transfer(r6 += 1): r6 = r6 -> r6 = (+ r6 1).
- At 2, the first path contributes r6 = (+ r6 1), r7 = r7.
- At 4, on the other path from 3, transfer(r7 = 10):
  r7 = r7 -> r7 = 10.
- At 5, transfer(r6 += 1): r6 = r6 -> r6 = (+ r6 1).
- At 2, join the two paths' contributions:
  join((+ r6 1), (+ r6 1)) = (+ r6 1) for r6;
  join(r7, 10) = (any r7 10) for r7.

Equal expressions stay unchanged; different expressions become ANY
alternatives. These summarize one iteration, without unrolling the loop.

Second phase:
-------------

The update r6 = (+ r6 1) says that each iteration adds 1 to r6.
Repeating this update n times gives r6_entry + n, a linear recurrence.

- At header 2, r6 = (+ r6 1) becomes (linear r6 1), meaning
  r6_entry + n, where n counts iterations from zero.
- At latch 6, substitute this recurrence into (+ r6 1):
  (+ (linear r6 1) 1) simplifies to (linear (+ r6 1) 1),
  meaning r6_entry + 1 + n.
- At both points, r7 retains (any r7 10): either its entry value
  or 10. ANY records alternatives, not numeric ranges or the path
  conditions selecting them. Unchanged variables retain their inputs.

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 include/linux/bpf_verifier.h |    7 +
 kernel/bpf/Makefile          |    2 +-
 kernel/bpf/scev.c            | 1640 ++++++++++++++++++++++++++++++++++++++++++
 kernel/bpf/verifier.c        |    9 +
 4 files changed, 1657 insertions(+), 1 deletion(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 963106328791..52f72e92eb0c 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -745,6 +745,7 @@ struct bpf_insn_aux_data {
 	bool call_with_percpu_alloc_ptr; /* {this,per}_cpu_ptr() with prog percpu alloc */
 	bool arena_scalar; /* ldx/stx/st/atomic through a number, it's an address in arena */
 	bool bb_end; /* last instruction of a basic block */
+	bool need_scev;
 	u8 alu_state; /* used in combination with alu_limit */
 	/* true if STX or LDX instruction is a part of a spill/fill
 	 * pattern for a bpf_fastcall call.
@@ -1000,6 +1001,7 @@ struct bpf_scc_info {
 };
 
 struct bpf_liveness;
+struct scev;
 
 struct bpf_fd_array {
 	union {
@@ -1164,6 +1166,7 @@ struct bpf_verifier_env {
 	struct bpf_iarray *succ;
 	struct bpf_iarray *gotox_tmp_buf;
 	int *idoms;
+	struct scev *scev;
 };
 
 static inline struct bpf_func_info_aux *subprog_aux(struct bpf_verifier_env *env, int subprog)
@@ -1964,4 +1967,8 @@ int bpf_compute_loops(struct bpf_verifier_env *env);
 int bpf_loop_at_index(struct bpf_verifier_env *env, u32 idx);
 bool bpf_is_nested_loop(struct bpf_verifier_env *env, int inner_header, int outer_header);
 
+int bpf_init_scev(struct bpf_verifier_env *env);
+void bpf_free_scev(struct bpf_verifier_env *env);
+int bpf_compute_scev(struct bpf_verifier_env *env);
+
 #endif /* _LINUX_BPF_VERIFIER_H */
diff --git a/kernel/bpf/Makefile b/kernel/bpf/Makefile
index 6210601eb40b..d61cbb59ad08 100644
--- a/kernel/bpf/Makefile
+++ b/kernel/bpf/Makefile
@@ -6,7 +6,7 @@ cflags-nogcse-$(CONFIG_X86)$(CONFIG_CC_IS_GCC) := -fno-gcse
 endif
 CFLAGS_core.o += -Wno-override-init $(cflags-nogcse-yy)
 
-obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o cnum.o log.o token.o liveness.o const_fold.o diagnostics.o loops.o
+obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o cnum.o log.o token.o liveness.o const_fold.o diagnostics.o loops.o scev.o
 obj-$(CONFIG_BPF_SYSCALL) += bpf_iter.o map_iter.o task_iter.o prog_iter.o link_iter.o
 obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o bloom_filter.o
 obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o bpf_insn_array.o
diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
new file mode 100644
index 000000000000..2d2a84156180
--- /dev/null
+++ b/kernel/bpf/scev.c
@@ -0,0 +1,1640 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */
+
+#include <linux/bpf_verifier.h>
+#include <linux/jhash.h>
+#include <linux/log2.h>
+#include <linux/min_heap.h>
+#include <linux/bug.h>
+
+#define REGS_NUM (MAX_BPF_REG + MAX_BPF_STACK_SLOTS)
+#define UNKNOWN_EXPR_ID 0
+#define OPAQUE_EXPR_ID  1
+
+/*
+ * BPF instructions use 'code', 'src_reg', 'off' and 'imm' fields for instruction encoding.
+ * For scalar evolution purpose we want to reuse most of the opcode definitions,
+ * but also add a few custom operations (REG and IMM).
+ * Use the enum below to uniformly represent all operations relevant for SCEV.
+ */
+enum expr_op {
+	/* leave range 0..255 for standard bpf opcodes */
+	UNKNOWN = 256, /* start custom opcodes from the second byte */
+	REG,
+	IMM,
+	SDIV, SMOD,
+	SEXT8, SEXT16, SEXT32,
+	ZEXT8, ZEXT16, ZEXT32,
+	BSWAP16, BSWAP32, BSWAP64,
+	/* during the loop body execution the value can be either of param[0] or param[1] */
+	ANY,
+	/* some value that verifier is not going to track precisely */
+	OPAQUE,
+	/* '*(u8/16/32 *)(r10 + X) = Y' writes define slot contents only partially */
+	SPILL8, SPILL16, SPILL32,
+	/*
+	 * SCEV expression corresponding to linear equation 'param[0] + param[1] * k',
+	 * where k is a loop iteration number. Loop here refers to innermost loop
+	 * containing instruction associated with this expression, as returned by
+	 * bpf_loop_at_index().
+	 */
+	LINEAR_SCEV,
+};
+
+struct expr {
+	u32 op;
+	union {
+		u32 params[2];
+		s64 imm;
+	};
+};
+
+struct expr_bucket {
+	u32 cnt;
+	u32 cap;
+	u32 ids[];
+};
+
+struct env {
+	bool empty;
+	u32 reg2expr[REGS_NUM];
+	u32 reg2scev[REGS_NUM];
+};
+
+struct insn_envs {
+	u32 cnt;
+	struct {
+		int loop_header;
+		struct env *env;
+	} entries[];
+};
+
+#define EXPR_STACK_DEPTH 8
+
+struct expr_stack_elt {
+	u32 id:28;
+	u32 pre:1;
+	u32 next_param:2;
+};
+
+DEFINE_MIN_HEAP(int, scev_worklist);
+
+struct scev {
+	/*
+	 * Expressions are identified by id, exprs_ht ensures that
+	 * each expression exists as a unique instance.
+	 * This allows for fast equivalence check: id1 === id2.
+	 */
+	struct expr_bucket **exprs_ht;
+	struct scev_worklist worklist;
+	struct insn_envs **envs;
+	struct expr *exprs;
+	/*
+	 * Loops are analyzed one by one, this array keeps track if a particular
+	 * instruction was visited on a current pass.
+	 */
+	u32 *discovered;
+	u32 exprs_ht_cnt;
+	int exprs_cnt;
+	int exprs_cap;
+	int envs_cnt;
+	int stack_sz;
+	struct expr_stack_elt expr_stack[EXPR_STACK_DEPTH];
+	u32 ids_buf[EXPR_STACK_DEPTH];
+};
+
+static u32 expr_hash(struct expr *e)
+{
+	return jhash_3words(e->op, e->params[0], e->params[1], 0);
+}
+
+static int expr_eq(struct expr *a, struct expr *b)
+{
+	return a->op == b->op && a->imm == b->imm;
+}
+
+static int add_expr(struct scev *scev, struct expr e)
+{
+	struct expr_bucket *bucket;
+	u32 i, id, hash, new_cap;
+	void *tmp;
+
+	hash = expr_hash(&e) & (scev->exprs_ht_cnt - 1);
+	bucket = scev->exprs_ht[hash];
+
+	if (bucket) {
+		for (i = 0; i < bucket->cnt; i++) {
+			id = bucket->ids[i];
+			if (expr_eq(&e, &scev->exprs[id]))
+				return id;
+		}
+	}
+
+	if (!bucket || bucket->cap == bucket->cnt) {
+		new_cap = bucket ? bucket->cap * 2 : 4;
+		bucket = kvrealloc(bucket, sizeof(*bucket) + sizeof(u32) * new_cap, GFP_KERNEL_ACCOUNT | __GFP_ZERO);
+		if (!bucket)
+			return -ENOMEM;
+		scev->exprs_ht[hash] = bucket;
+		bucket->cap = new_cap;
+	}
+
+	if (scev->exprs_cnt == scev->exprs_cap) {
+		new_cap = scev->exprs_cap ? scev->exprs_cap * 2 : 256;
+		tmp = kvrealloc(scev->exprs, sizeof(struct expr) * new_cap, GFP_KERNEL_ACCOUNT);
+		if (!tmp)
+			return -ENOMEM;
+		scev->exprs = tmp;
+		scev->exprs_cap = new_cap;
+	}
+
+	id = scev->exprs_cnt++;
+	scev->exprs[id] = e;
+	bucket->ids[bucket->cnt++] = id;
+	return id;
+}
+
+static int expr2(struct scev *scev, u32 op, int a, int b)
+{
+	if (a < 0)
+		return a;
+	if (b < 0)
+		return b;
+	return add_expr(scev, (struct expr){ .op = op, .params = {a, b} });
+}
+
+static int expr1(struct scev *scev, u32 op, int a)
+{
+	if (a < 0)
+		return a;
+	return expr2(scev, op, a, 0);
+}
+
+static int expr0(struct scev *scev, u32 op)
+{
+	return expr2(scev, op, 0, 0);
+}
+
+static int imm_expr(struct scev *scev, s64 value)
+{
+	return add_expr(scev, (struct expr){ .op = IMM, .imm = value });
+}
+
+static bool same_exprs(struct scev *scev, int id_a, int id_b)
+{
+	return id_a == id_b;
+}
+
+static bool is_imm(struct scev *scev, int id, s64 *imm)
+{
+	if (scev->exprs[id].op != IMM)
+		return false;
+	*imm = scev->exprs[id].imm;
+	return true;
+}
+
+static bool is_op(struct scev *scev, int id, u32 op)
+{
+	if (scev->exprs[id].op != op)
+		return false;
+	return true;
+}
+
+static bool is_unop(struct scev *scev, enum expr_op op, int id, u32 *p0)
+{
+	if (scev->exprs[id].op != op)
+		return false;
+	*p0 = scev->exprs[id].params[0];
+	return true;
+}
+
+static bool is_binop(struct scev *scev, enum expr_op op, int id, u32 *p0, u32 *p1)
+{
+	if (scev->exprs[id].op != op)
+		return false;
+	*p0 = scev->exprs[id].params[0];
+	*p1 = scev->exprs[id].params[1];
+	return true;
+}
+
+static bool is_reg(struct scev *scev, int id, u32 *reg)
+{
+	return is_unop(scev, REG, id, reg);
+}
+
+static bool is_add(struct scev *scev, int id, u32 *left, u32 *right)
+{
+	return is_binop(scev, BPF_ADD, id, left, right);
+}
+
+static bool is_zext32(struct scev *scev, int id, u32 *left)
+{
+	return is_unop(scev, ZEXT32, id, left);
+}
+
+static bool is_any(struct scev *scev, int id, u32 *left, u32 *right)
+{
+	return is_binop(scev, ANY, id, left, right);
+}
+
+static bool is_linear(struct scev *scev, int id, u32 *base, u32 *slope)
+{
+	return is_binop(scev, LINEAR_SCEV, id, base, slope);
+}
+
+static bool is_opaque(struct scev *scev, int id)
+{
+	return is_op(scev, id, OPAQUE);
+}
+
+static void log_reg(struct bpf_verifier_env *env, u32 reg)
+{
+	if (reg < MAX_BPF_REG)
+		bpf_log(&env->log, "r%d", reg);
+	else
+		bpf_log(&env->log, "*fp%d", (MAX_BPF_REG - reg - 1) * 8);
+}
+
+static const char *op_str(u32 op)
+{
+	switch (op) {
+	case BPF_ADD:  return "+";
+	case BPF_SUB:  return "-";
+	case BPF_MUL:  return "*";
+	case BPF_DIV:  return "/";
+	case SDIV:     return "s/";
+	case BPF_OR:   return "|";
+	case BPF_AND:  return "&";
+	case BPF_LSH:  return "<<";
+	case BPF_RSH:  return ">>";
+	case BPF_NEG:  return "-";
+	case BPF_MOD:  return "%";
+	case SMOD:     return "s%";
+	case BPF_XOR:  return "^";
+	case BPF_ARSH: return "s>>";
+	case SEXT8:    return "sext8";
+	case SEXT16:   return "sext16";
+	case SEXT32:   return "sext32";
+	case ZEXT8:    return "zext8";
+	case ZEXT16:   return "zext16";
+	case ZEXT32:   return "zext32";
+	case BSWAP16:  return "bswap16";
+	case BSWAP32:  return "bswap32";
+	case BSWAP64:  return "bswap64";
+	case SPILL8:   return "spill8";
+	case SPILL16:  return "spill16";
+	case SPILL32:  return "spill32";
+	case ANY:      return "any";
+	case LINEAR_SCEV:  return "linear";
+	}
+	return NULL;
+}
+
+static u32 op_params_num(u32 op)
+{
+	switch (op) {
+	case BPF_ADD:
+	case BPF_SUB:
+	case BPF_MUL:
+	case BPF_DIV:
+	case BPF_MOD:
+	case BPF_OR:
+	case BPF_XOR:
+	case BPF_AND:
+	case BPF_LSH:
+	case BPF_RSH:
+	case BPF_ARSH:
+	case SDIV:
+	case SMOD:
+	case ANY:
+	case LINEAR_SCEV:
+		return 2;
+	case BPF_NEG:
+	case SEXT8:
+	case SEXT16:
+	case SEXT32:
+	case ZEXT8:
+	case ZEXT16:
+	case ZEXT32:
+	case BSWAP16:
+	case BSWAP32:
+	case BSWAP64:
+	case SPILL8:
+	case SPILL16:
+	case SPILL32:
+		return 1;
+	case REG:
+	case IMM:
+	case OPAQUE:
+		return 0;
+	default:
+		return 0;
+	}
+}
+
+static bool expr_stack_push(struct scev *scev, u32 id)
+{
+	if (scev->stack_sz >= EXPR_STACK_DEPTH)
+		return false;
+	scev->expr_stack[scev->stack_sz].id = id;
+	scev->expr_stack[scev->stack_sz].pre = true;
+	scev->expr_stack[scev->stack_sz].next_param = 0;
+	scev->stack_sz++;
+	return true;
+}
+
+enum {
+	PRE = BIT(1), POST = BIT(2), DEPTH_LIMIT = BIT(3)
+};
+
+static bool expr_next(struct scev *scev, u32 *id, u32 *order)
+{
+	struct expr_stack_elt *elt;
+	struct expr *expr;
+	u32 num_params;
+
+	if (scev->stack_sz == 0)
+		return false;
+
+	elt = &scev->expr_stack[scev->stack_sz - 1];
+	*id = elt->id;
+	*order = 0;
+	expr = &scev->exprs[elt->id];
+	num_params = op_params_num(expr->op);
+	if (elt->pre) {
+		elt->pre = false;
+		*order = PRE;
+		return true;
+	}
+	if (elt->next_param == num_params) {
+		*order = POST;
+		scev->stack_sz--;
+		return true;
+	}
+	if (scev->stack_sz == EXPR_STACK_DEPTH) {
+		*order = POST | DEPTH_LIMIT;
+		scev->stack_sz--;
+		return true;
+	}
+	expr_stack_push(scev, expr->params[elt->next_param]);
+	elt->next_param++;
+	return expr_next(scev, id, order);
+}
+
+static void log_expr(struct bpf_verifier_env *env, u32 id)
+{
+	struct bpf_verifier_log *log = &env->log;
+	struct scev *scev = env->scev;
+	struct expr *expr;
+	const char *str;
+	u32 order;
+
+	scev->stack_sz = 0;
+	expr_stack_push(scev, id);
+	while (expr_next(scev, &id, &order)) {
+		if ((order & PRE) && scev->stack_sz > 1)
+			bpf_log(log, " ");
+		expr = &scev->exprs[id];
+		switch (expr->op) {
+		case UNKNOWN:
+			if (order & PRE)
+				bpf_log(log, "?");
+			break;
+		case OPAQUE:
+			if (order & PRE)
+				bpf_log(log, "_");
+			break;
+		case REG:
+			if (order & PRE)
+				log_reg(env, expr->params[0]);
+			break;
+		case IMM:
+			if (order & PRE)
+				bpf_log(log, "%lld", expr->imm);
+			break;
+		default:
+			if (order & PRE) {
+				str = op_str(expr->op);
+				bpf_log(log, "(");
+				if (str)
+					bpf_log(log, "%s", str);
+				else
+					bpf_log(log, "bad-expr-op %x", expr->op);
+			}
+			if (order & DEPTH_LIMIT)
+				bpf_log(log, "...");
+			if (order & POST)
+				bpf_log(log, ")");
+		}
+	}
+}
+
+static bool reg_alive_at(struct bpf_verifier_env *env, u32 reg, u32 insn_idx)
+{
+	struct bpf_insn_aux_data *aux = env->insn_aux_data;
+	u16 live_regs = aux[insn_idx].live_regs_before;
+
+	return reg < MAX_BPF_REG
+	       ? (BIT(reg) & live_regs)
+	       : test_bit(reg - MAX_BPF_REG, aux[insn_idx].live_stack_before);
+}
+
+static void print_env(struct bpf_verifier_env *env, struct env *e, u32 insn_idx)
+{
+	struct bpf_verifier_log *log = &env->log;
+	struct scev *scev = env->scev;
+	bool printed_some = false;
+	bool is_self_scev;
+	bool is_self_reg;
+	int i, r;
+
+	if (e->empty) {
+		bpf_log(log, "  <empty>\n");
+		return;
+	}
+
+	for (i = 0; i < REGS_NUM; i++) {
+		if (!reg_alive_at(env, i, insn_idx))
+			continue;
+		is_self_reg = is_reg(scev, e->reg2expr[i], &r) && i == r;
+		is_self_scev = is_reg(scev, e->reg2scev[i], &r) && i == r;
+		if (is_self_reg && is_self_scev)
+			continue;
+		printed_some = true;
+		bpf_log(log, "  ");
+		log_reg(env, i);
+		bpf_log(log, "=");
+		log_expr(env, e->reg2expr[i]);
+		if (e->reg2expr[i] != UNKNOWN_EXPR_ID) {
+			bpf_log(log, " / ");
+			log_expr(env, e->reg2scev[i]);
+		}
+		bpf_log(log, "\n");
+	}
+	if (!printed_some)
+		bpf_log(log, "  <all regs unchanged>\n");
+}
+static struct env *find_loop_env(struct scev *scev, int loop_header, int insn_idx)
+{
+	struct insn_envs *envs = scev->envs[insn_idx];
+	u32 i;
+
+	if (!envs)
+		return NULL;
+	for (i = 0; i < envs->cnt; i++)
+		if (envs->entries[i].loop_header == loop_header)
+			return envs->entries[i].env;
+	return NULL;
+}
+
+static struct env *find_header_env(struct scev *scev, int header)
+{
+	return find_loop_env(scev, header, header);
+}
+
+static struct env *get_loop_env(struct scev *scev, int loop_header, int insn_idx)
+{
+	struct insn_envs *envs = scev->envs[insn_idx];
+	u32 cnt = envs ? envs->cnt : 0;
+	struct insn_envs *tmp;
+	struct env *e;
+
+	e = find_loop_env(scev, loop_header, insn_idx);
+	if (e)
+		return e;
+
+	e = kzalloc(sizeof(*e), GFP_KERNEL_ACCOUNT);
+	if (!e)
+		return NULL;
+
+	tmp = krealloc(envs, struct_size(envs, entries, cnt + 1), GFP_KERNEL_ACCOUNT);
+	if (!tmp) {
+		kfree(e);
+		return NULL;
+	}
+
+	e->empty = true;
+	tmp->entries[cnt].loop_header = loop_header;
+	tmp->entries[cnt].env = e;
+	tmp->cnt = cnt + 1;
+	scev->envs[insn_idx] = tmp;
+	return e;
+}
+
+static int replace_reg(struct scev *scev, struct env *e, u32 reg, int id)
+{
+	if (id < 0)
+		return id;
+	e->reg2expr[reg] = id;
+	return 0;
+}
+
+static int setup_initial_loop_env(struct bpf_verifier_env *env, struct env *e, int insn_idx)
+{
+	struct scev *scev = env->scev;
+	int i, err;
+
+	/*
+	 * Initial assignments are of form rX -> rX are needed as seeds for subsequent
+	 * work queue pop / transfer() calls. Because e->empty is true, the first backedge
+	 * join() will overwrite the them. compute_header_scevs() accounts for register
+	 * states at loop entry.
+	 */
+	for (i = 0; i < REGS_NUM; i++) {
+		err = replace_reg(scev, e, i, expr1(scev, REG, i));
+		if (err)
+			return err;
+	}
+	return 0;
+}
+
+static void forget_call_regs(struct env *e)
+{
+	int i;
+
+	for (i = BPF_REG_0; i <= BPF_REG_5; i++)
+		e->reg2expr[i] = UNKNOWN_EXPR_ID;
+}
+
+static u32 spill_spi(struct bpf_insn *insn)
+{
+	return -insn->off / BPF_REG_SIZE - 1;
+}
+
+static u32 off_to_reg(struct bpf_insn *insn)
+{
+	return spill_spi(insn) + MAX_BPF_REG;
+}
+
+static bool is_spill_off(int off)
+{
+	return off % BPF_REG_SIZE == 0 &&
+	       off <= -BPF_REG_SIZE &&
+	       off >= -MAX_BPF_STACK_JIT;
+}
+
+static void mark_opaque(struct env *e, int off, int size)
+{
+	int b, spi;
+
+	for (b = off; b < off + size; b++) {
+		if (b >= 0 || b < -MAX_BPF_STACK_JIT)
+			continue;
+		spi = (-b - 1) / BPF_REG_SIZE;
+		e->reg2expr[MAX_BPF_REG + spi] = OPAQUE_EXPR_ID;
+	}
+}
+
+static int mk_spill(struct scev *scev, u8 size, int id)
+{
+	switch (size) {
+	case BPF_B:  return expr1(scev, SPILL8,  id);
+	case BPF_H:  return expr1(scev, SPILL16, id);
+	case BPF_W:  return expr1(scev, SPILL32, id);
+	case BPF_DW: return id;
+	}
+	return UNKNOWN_EXPR_ID;
+}
+
+static int mk_fill(struct scev *scev, int id, u8 code)
+{
+	bool sx = BPF_MODE(code) == BPF_MEMSX;
+
+#ifdef __BIG_ENDIAN
+	/* A narrow fill from a DW spill reads high bits on big-endian. */
+	if (BPF_SIZE(code) != BPF_DW)
+		return OPAQUE_EXPR_ID;
+#endif
+
+	switch (BPF_SIZE(code)) {
+	case BPF_B:  return expr1(scev, sx ? SEXT8  : ZEXT8,  id);
+	case BPF_H:  return expr1(scev, sx ? SEXT16 : ZEXT16, id);
+	case BPF_W:  return expr1(scev, sx ? SEXT32 : ZEXT32, id);
+	case BPF_DW: return sx ? UNKNOWN_EXPR_ID : id; /* DW sign-extended load is invalid */
+	}
+	return UNKNOWN_EXPR_ID;
+}
+
+static int maybe_store_fp(struct scev *scev, struct env *e, struct bpf_insn *insn, int id)
+{
+	u8 size = BPF_SIZE(insn->code);
+
+	if (insn->dst_reg != BPF_REG_FP)
+		return 0;
+	if (is_spill_off(insn->off))
+		return replace_reg(scev, e, off_to_reg(insn), mk_spill(scev, size, id));
+	mark_opaque(e, insn->off, bpf_size_to_bytes(size));
+	return 0;
+}
+
+static int maybe_load_fp(struct scev *scev, struct env *e, struct bpf_insn *insn)
+{
+	int id;
+
+	if (insn->src_reg == BPF_REG_FP && is_spill_off(insn->off))
+		id = mk_fill(scev, e->reg2expr[off_to_reg(insn)], insn->code);
+	else
+		id = OPAQUE_EXPR_ID;
+	return replace_reg(scev, e, insn->dst_reg, id);
+}
+
+static int transfer(struct bpf_verifier_env *env, struct env *e, int idx)
+{
+	const bool little_endian = htons(0x3412) == 0x1234;
+	struct bpf_insn *insn = &env->prog->insnsi[idx];
+	struct scev *scev = env->scev;
+	u32 *reg2expr = e->reg2expr;
+	u8 class = BPF_CLASS(insn->code);
+	u8 x_or_k = BPF_SRC(insn->code);
+	u8 opcode = BPF_OP(insn->code);
+	u8 mode = BPF_MODE(insn->code);
+	u32 dst = insn->dst_reg;
+	u32 src = insn->src_reg;
+	u32 op, sext;
+	bool need_bswap;
+	int i, id;
+	s64 imm;
+
+	switch (class) {
+	case BPF_ALU:
+	case BPF_ALU64:
+		switch (opcode) {
+		case BPF_MOV:
+			switch (insn->off) {
+			case 0: sext = 0; break;
+			case 8: sext = SEXT8; break;
+			case 16: sext = SEXT16; break;
+			case 32: sext = SEXT32; break;
+			default:
+				goto mark_dst_unknown;
+			}
+
+			if (x_or_k == BPF_X && insn->imm == 0)
+				id = reg2expr[src];
+			else if (x_or_k == BPF_K && src == 0 && insn->off == 0)
+				id = imm_expr(scev, insn->imm);
+			else
+				goto mark_dst_unknown;
+
+			if (sext)
+				id = expr1(scev, sext, id);
+			if (class == BPF_ALU)
+				id = expr1(scev, ZEXT32, id);
+
+			return replace_reg(scev, e, dst, id);
+
+		case BPF_ADD:
+		case BPF_SUB:
+		case BPF_MUL:
+		case BPF_DIV:
+		case BPF_MOD:
+		case BPF_OR:
+		case BPF_XOR:
+		case BPF_AND:
+		case BPF_LSH:
+		case BPF_RSH:
+		case BPF_ARSH:
+			if (opcode == BPF_DIV && insn->off == 1)
+				op = SDIV;
+			else if (opcode == BPF_MOD && insn->off == 1)
+				op = SMOD;
+			else if (insn->off == 0)
+				op = opcode;
+			else
+				goto mark_dst_unknown;
+
+			if (x_or_k == BPF_X && insn->imm == 0)
+				id = reg2expr[src];
+			else if (x_or_k == BPF_K && src == 0)
+				id = imm_expr(scev, insn->imm);
+			else
+				goto mark_dst_unknown;
+
+			id = expr2(scev, op, reg2expr[dst], id);
+
+			if (class == BPF_ALU)
+				id = expr1(scev, ZEXT32, id);
+
+			return replace_reg(scev, e, dst, id);
+
+		case BPF_NEG:
+			if (src == 0 && insn->off == 0 && insn->imm == 0)
+				id = expr1(scev, BPF_NEG, reg2expr[dst]);
+			else
+				goto mark_dst_unknown;
+
+			if (class == BPF_ALU)
+				id = expr1(scev, ZEXT32, id);
+
+			return replace_reg(scev, e, dst, id);
+
+		case BPF_END:
+			if (src != 0 || insn->off != 0 || (class == BPF_ALU64 && x_or_k != 0))
+				goto mark_dst_unknown;
+
+			need_bswap = class == BPF_ALU64 || ((x_or_k == BPF_TO_LE) != little_endian);
+			switch (insn->imm) {
+			case 16: op = need_bswap ? BSWAP16 : ZEXT16; break;
+			case 32: op = need_bswap ? BSWAP32 : ZEXT32; break;
+			case 64: op = need_bswap ? BSWAP64 : 0; break;
+			default:
+				goto mark_dst_unknown;
+			}
+
+			id = reg2expr[dst];
+			if (op)
+				id = expr1(scev, op, reg2expr[dst]);
+
+			return replace_reg(scev, e, dst, id);
+		default:
+			goto mark_dst_unknown;
+		}
+		break;
+	case BPF_LDX:
+		switch (mode) {
+		case BPF_MEM:
+		case BPF_MEMSX:
+			return maybe_load_fp(scev, e, insn);
+		default:
+			goto mark_dst_unknown;
+		}
+	case BPF_STX:
+		switch (mode) {
+		case BPF_MEM:
+			return maybe_store_fp(scev, e, insn, reg2expr[src]);
+		case BPF_ATOMIC:
+			if (insn->imm == BPF_LOAD_ACQ)
+				return maybe_load_fp(scev, e, insn);
+			if (insn->imm == BPF_STORE_REL) {
+				return maybe_store_fp(scev, e, insn, reg2expr[src]);
+			}
+			/*
+			 * verifier does not track other atomic ops precisely,
+			 * hence mark the results as opaque.
+			 */
+			if (insn->dst_reg == BPF_REG_FP)
+				mark_opaque(e, insn->off, bpf_size_to_bytes(BPF_SIZE(insn->code)));
+			if (insn->imm == BPF_CMPXCHG)
+				return replace_reg(scev, e, BPF_REG_0, OPAQUE_EXPR_ID);
+			if (insn->imm & BPF_FETCH)
+				return replace_reg(scev, e, src, OPAQUE_EXPR_ID);
+			break;
+		}
+		break;
+	case BPF_ST:
+		if (insn->dst_reg == BPF_REG_FP) {
+			id = imm_expr(scev, insn->imm);
+			if (id < 0)
+				return id;
+			return maybe_store_fp(scev, e, insn, id);
+		}
+		break;
+	case BPF_JMP:
+	case BPF_JMP32:
+		if (opcode == BPF_CALL)
+			forget_call_regs(e);
+		/* for non-CALL there are no changes in register states */
+		break;
+	case BPF_LD:
+		switch (mode) {
+		case BPF_IMM:
+			/* rX = imm ll */
+			if (BPF_SIZE(insn->code) == BPF_DW && insn->src_reg == 0) {
+				imm = ((u64)(insn + 1)->imm << 32) | (u32)insn->imm;
+				return replace_reg(scev, e, dst, imm_expr(scev, imm));
+			}
+			/* map, map value, BTF id, function */
+			if (BPF_SIZE(insn->code) == BPF_DW)
+				return replace_reg(scev, e, dst, OPAQUE_EXPR_ID);
+			goto mark_dst_unknown;
+		case BPF_ABS:
+		case BPF_IND:
+			forget_call_regs(e);
+			break;
+		default:
+			goto mark_dst_unknown;
+		}
+		break;
+	default:
+		/* unknown instruction, nuke state */
+		for (i = 0; i < REGS_NUM; i++)
+			reg2expr[i] = UNKNOWN_EXPR_ID;
+		break;
+	}
+	return 0;
+
+mark_dst_unknown:
+	reg2expr[dst] = UNKNOWN_EXPR_ID;
+	return 0;
+}
+
+/*
+ * Construct minimal 'ANY' expression by traversing 'a' and 'b'
+ * and accumulating non-duplicated non-ANY entries.
+ */
+static int mk_any(struct scev *scev, int a, int b)
+{
+	struct expr_stack_elt *elt;
+	struct expr *expr;
+	u32 i, j, ids_buf_sz;
+	u32 roots[2] = {a, b};
+	int id;
+
+	if (a < 0)
+		return a;
+	if (b < 0)
+		return b;
+	ids_buf_sz = 0;
+	for (i = 0; i < ARRAY_SIZE(roots); i++) {
+		scev->stack_sz = 0;
+		expr_stack_push(scev, roots[i]);
+		while (scev->stack_sz) {
+			elt = &scev->expr_stack[--scev->stack_sz];
+			id = elt->id;
+			expr = &scev->exprs[elt->id];
+			if (expr->op == ANY) {
+				if (!expr_stack_push(scev, expr->params[1]) ||
+				    !expr_stack_push(scev, expr->params[0]))
+					return UNKNOWN_EXPR_ID;
+			} else {
+				for (j = 0; j < ids_buf_sz; j++) {
+					if (same_exprs(scev, scev->ids_buf[j], id))
+						goto next;
+				}
+				if (ids_buf_sz == ARRAY_SIZE(scev->ids_buf))
+					return UNKNOWN_EXPR_ID;
+				scev->ids_buf[ids_buf_sz++] = id;
+			}
+next:;
+		}
+	}
+	if (WARN_ON(ids_buf_sz == 0))
+		return -EFAULT;
+	id = scev->ids_buf[0];
+	for (i = 1; i < ids_buf_sz; i++) {
+		id = expr2(scev, ANY, id, scev->ids_buf[i]);
+		if (id < 0)
+			return id;
+	}
+	return id;
+}
+
+static int join(struct scev *scev, struct env *acc, struct env *cur)
+{
+	int i, id;
+
+	if (acc->empty) {
+		memcpy(acc, cur, sizeof(*acc));
+		acc->empty = false;
+		return 0;
+	}
+
+	for (i = 0; i < REGS_NUM; i++) {
+		if (!same_exprs(scev, acc->reg2expr[i], cur->reg2expr[i])) {
+			id = mk_any(scev, acc->reg2expr[i], cur->reg2expr[i]);
+			if (id < 0)
+				return id;
+			acc->reg2expr[i] = id;
+		}
+	}
+	return 0;
+}
+
+/* Mark any register modified in 'e' as unknown in 'acc'. */
+static void forget_modified_regs(struct scev *scev, struct env *acc, struct env *e)
+{
+	u32 i, reg;
+
+	for (i = 0; i < REGS_NUM; i++) {
+		if (e && is_reg(scev, e->reg2expr[i], &reg) && reg == i)
+			continue;
+		acc->reg2expr[i] = UNKNOWN_EXPR_ID;
+	}
+}
+
+/* For a loop H at `header`, mark any register modified as a result of H execution as unknown. */
+static void forget_non_invariants(struct bpf_verifier_env *env, struct env *acc, int header)
+{
+	struct bpf_insn_aux_data *aux = env->insn_aux_data;
+	struct bpf_loop *loop = aux[header].loop;
+	struct scev *scev = env->scev;
+	int i, idx, h;
+
+	acc->empty = false;
+	forget_modified_regs(scev, acc, find_header_env(scev, header));
+	for (i = 0; i < loop->exits_cnt; i++) {
+		idx = loop->exits[i].from;
+		h = bpf_loop_at_index(env, idx);
+		/*
+		 * In order to account for situations like this:
+		 *
+		 * header:
+		 *   do {
+		 *	saved = rX;
+		 *	rX = -1000;
+		 *	do {				// H
+		 *		if (random())
+		 *			goto out;	// E
+		 *	} while (inner_again);
+		 *	rX = saved;
+		 *    } while (outer_again);
+		 *  out:
+		 *
+		 * For exits originating in nested loops check that rX is invariant
+		 * both at the header (H) and at exit (E).
+		 */
+		while (h != header) {
+			forget_modified_regs(scev, acc, find_header_env(scev, h));
+			forget_modified_regs(scev, acc, find_loop_env(scev, h, idx));
+			idx = h;
+			h = aux[h].loop_header;
+		}
+		/*
+		 * For exits originating at the same loop the header is already accounted for,
+		 * check only the exit environment.
+		 */
+		forget_modified_regs(scev, acc, find_loop_env(scev, header, idx));
+	}
+}
+
+/* Order worklist in reverse post-order. */
+static bool reverse_postorder_less(const void *a, const void *b, void *arg)
+{
+	const int *rank = arg;
+
+	return rank[*(const int *)a] > rank[*(const int *)b];
+}
+
+static const struct min_heap_callbacks worklist_cb = {
+	.less = reverse_postorder_less,
+};
+
+static int worklist_push(struct bpf_verifier_env *env, int loop_header, int idx)
+{
+	struct scev *scev = env->scev;
+	struct scev_worklist *worklist = &scev->worklist;
+	u32 tag = (u32)loop_header + 1;
+
+	if (scev->discovered[idx] == tag)
+		return 0;
+
+	if (min_heap_full_inline(worklist)) {
+		size_t size = worklist->size ? worklist->size * 2 : 16;
+		int *data;
+
+		data = kvrealloc(worklist->data, array_size(size, sizeof(*data)),
+				GFP_KERNEL_ACCOUNT);
+		if (!data)
+			return -ENOMEM;
+		worklist->data = data;
+		worklist->size = size;
+	}
+	if (!min_heap_push_inline(worklist, &idx, &worklist_cb, env->cfg.postorder_nums))
+		return -EFAULT;
+
+	scev->discovered[idx] = tag;
+	return 0;
+}
+
+static bool reg_alive_at_succ(struct bpf_verifier_env *env, u32 r, u32 insn_idx)
+{
+	struct bpf_iarray *succ = bpf_insn_successors(env, insn_idx);
+	u32 i;
+
+	for (i = 0; i < succ->cnt; i++)
+		if (reg_alive_at(env, r, succ->items[i]))
+			return true;
+	return false;
+}
+
+enum changes_at {
+	LOG_AT_TRANSFER,
+	LOG_AT_JOIN,
+};
+
+static void log_env_changes(struct bpf_verifier_env *env, enum changes_at at,
+			    struct env *old, struct env *new, int insn_idx)
+{
+	struct bpf_verifier_log *log = &env->log;
+	struct scev *scev = env->scev;
+	u64 null_pos = log->end_pos;
+	u32 old_id, new_id, regno;
+	bool any_changes = false;
+	bool is_self_scev;
+	u64 len;
+	int r;
+
+	bpf_log(log, "%s %4d: ", at == LOG_AT_TRANSFER ? "t" : "j", insn_idx);
+	bpf_verbose_insn(env, &env->prog->insnsi[insn_idx]);
+	len = log->end_pos - null_pos;
+	bpf_log(log, "%*s", max(37 - (int)len, 1), " ");
+	bpf_log(log, " ; ");
+	for (r = 0; r < REGS_NUM; r++) {
+		old_id = old->reg2expr[r];
+		new_id = new->reg2expr[r];
+		if (at == LOG_AT_TRANSFER && !reg_alive_at_succ(env, r, insn_idx))
+			continue;
+		if (at == LOG_AT_JOIN && !reg_alive_at(env, r, insn_idx))
+			continue;
+		if (same_exprs(scev, old_id, new_id))
+			continue;
+		is_self_scev = is_reg(scev, new_id, &regno) && r == regno;
+		if (old->empty && is_self_scev)
+			continue;
+		if (any_changes)
+			bpf_log(log, ", ");
+		log_reg(env, r);
+		bpf_log(log, " ");
+		log_expr(env, old_id);
+		bpf_log(log, " -> ");
+		log_expr(env, new_id);
+		any_changes = true;
+	}
+	if (old->empty && !any_changes)
+		bpf_log(log, "<empty join>");
+	bpf_log(log, "\n");
+	if (!old->empty && !any_changes)
+		bpf_vlog_reset(log, null_pos);
+}
+
+static bool is_probe_read_helper(u32 func_id)
+{
+	return func_id == BPF_FUNC_probe_read ||
+	       func_id == BPF_FUNC_probe_read_kernel ||
+	       func_id == BPF_FUNC_probe_read_user ||
+	       func_id == BPF_FUNC_probe_read_str ||
+	       func_id == BPF_FUNC_probe_read_kernel_str ||
+	       func_id == BPF_FUNC_probe_read_user_str;
+}
+
+/*
+ * If instruction is an indirect write to stack, invalidate SCEVs for spi's
+ * that this instruction can touch.
+ */
+static void reset_scevs_at_indirect_writes(struct bpf_verifier_env *env, struct env *cur_env, int idx)
+{
+	struct bpf_insn *insn = &env->prog->insnsi[idx];
+	struct scev *scev = env->scev;
+	u8 class = BPF_CLASS(insn->code);
+	const unsigned long *mask;
+	u32 spi, reg;
+	bool opaque;
+
+	/* Direct fp stores are fine. */
+	if ((class == BPF_ST || class == BPF_STX) && insn->dst_reg == BPF_REG_FP)
+		return;
+
+	mask = bpf_may_write_mask(env, idx);
+	opaque = bpf_helper_call(insn) && is_probe_read_helper(insn->imm);
+	for_each_set_bit(spi, mask, MAX_BPF_STACK_SLOTS) {
+		reg = MAX_BPF_REG + spi;
+		if (cur_env->reg2expr[reg] == UNKNOWN_EXPR_ID)
+			continue;
+		replace_reg(scev, cur_env, reg, opaque ? OPAQUE_EXPR_ID : UNKNOWN_EXPR_ID);
+	}
+}
+
+/* Find the topmost loop header containing idx inside cur_header, or -1 if none. */
+static int topmost_nested_loop(struct bpf_verifier_env *env, int idx, int cur_header)
+{
+	struct bpf_insn_aux_data *aux = env->insn_aux_data;
+	int header;
+
+	for (header = bpf_loop_at_index(env, idx); header >= 0;
+	     header = aux[header].loop_header)
+		if (aux[header].loop_header == cur_header)
+			return header;
+
+	return -1;
+}
+
+/*
+ * Join cur_env into the successor's environment and schedule their traversal.
+ * Map successors in nested loops to their topmost nested header.
+ * Ignore successors outside the current loop and its nested loops.
+ */
+static int join_successor(struct bpf_verifier_env *env, int cur_header, int succ_idx,
+			  struct env *old_env, struct env *cur_env)
+{
+	bool log_level2 = env->log.level & BPF_LOG_LEVEL2;
+	struct scev *scev = env->scev;
+	int err, nested_header;
+	struct env *succ_env;
+
+	/*
+	 * There are several possibilities for a successor:
+	 * - succ_idx can be a part of a loop outside of the cur_header's loop,
+	 *   such edges are ignored.
+	 * - succ_idx can be a part of the same loop as cur_header,
+	 *   for such edges succ_idx environment is updated:
+	 *     e[succ_idx] = join(e[succ_idx], cur_env)
+	 * - succ_idx can be a part of some loop inner to cur_header,
+	 *   in such a case there exists some loop header H,
+	 *   such that H.loop_header == cur_header
+	 *   and H is the same as succ_idx's loop or contains it.
+	 */
+	if (bpf_loop_at_index(env, succ_idx) != cur_header) {
+		/*
+		 * topmost_nested_loop() either finds H or returns -1,
+		 * in case if succ_idx is a part of a loop outer to cur_header.
+		 */
+		nested_header = topmost_nested_loop(env, succ_idx, cur_header);
+		if (nested_header < 0)
+			return 0;
+		succ_idx = nested_header;
+	}
+	succ_env = get_loop_env(scev, cur_header, succ_idx);
+	if (!succ_env)
+		return -ENOMEM;
+	if (log_level2)
+		memcpy(old_env, succ_env, sizeof(*old_env));
+	err = join(scev, succ_env, cur_env);
+	if (err)
+		return err;
+	err = worklist_push(env, cur_header, succ_idx);
+	if (err)
+		return err;
+	if (log_level2)
+		log_env_changes(env, LOG_AT_JOIN, old_env, succ_env, succ_idx);
+	return 0;
+}
+
+/* Unsupported loops get only an unknown header environment. */
+static bool can_compute_loop_scev(const struct bpf_loop *loop)
+{
+	return !loop->irreducible && !loop->exits_overflow;
+}
+
+static int compute_scev_for_loop(struct bpf_verifier_env *env, int cur_header)
+{
+	struct scev_worklist *worklist = &env->scev->worklist;
+	struct bpf_insn_aux_data *aux = env->insn_aux_data;
+	struct bpf_loop *cur_loop = aux[cur_header].loop;
+	struct env *header_env;
+	struct scev *scev = env->scev;
+	struct bpf_loop *nested_loop;
+	struct env *cur_env = NULL;
+	struct env *old_env = NULL;
+	struct bpf_iarray *succ;
+	bool log_level2 = env->log.level & BPF_LOG_LEVEL2;
+	int s, i, err, idx, succ_idx;
+
+	if (log_level2)
+		bpf_log(&env->log, "Computing SCEV for loop at %d:\n", cur_header);
+
+	/*
+	 * For irreducible loops, and for loops nesting a loop with a truncated
+	 * exit list, just assume that everything is clobbered for now.
+	 */
+	if (!can_compute_loop_scev(cur_loop)) {
+		header_env = get_loop_env(scev, cur_header, cur_header);
+		if (!header_env)
+			return -ENOMEM;
+		/* The freshly allocated environment has all expressions unknown. */
+		header_env->empty = false;
+		return 0;
+	}
+
+	cur_env = kzalloc(sizeof(*cur_env), GFP_KERNEL_ACCOUNT);
+	if (!cur_env)
+		goto nomem;
+	if (log_level2) {
+		old_env = kzalloc(sizeof(*old_env), GFP_KERNEL_ACCOUNT);
+		if (!old_env)
+			goto nomem;
+	}
+	header_env = get_loop_env(scev, cur_header, cur_header);
+	if (!header_env)
+		goto nomem;
+	err = setup_initial_loop_env(env, header_env, cur_header);
+	if (err)
+		goto out;
+	err = worklist_push(env, cur_header, cur_header);
+	if (err)
+		goto out;
+
+	while (worklist->nr) {
+		idx = *min_heap_peek_inline(worklist);
+		min_heap_pop_inline(worklist, &worklist_cb, env->cfg.postorder_nums);
+
+		/* join_successor() maps nested-loop entries to their representative header. */
+		nested_loop = idx != cur_header ? aux[idx].loop : NULL;
+		memcpy(cur_env, find_loop_env(scev, cur_header, idx), sizeof(*cur_env));
+		if (nested_loop) {
+			/*
+			 * Process nested loop as a single instruction by
+			 * forgetting anything non-invariant in the nested loop
+			 */
+			if (log_level2)
+				memcpy(old_env, cur_env, sizeof(*old_env));
+			forget_non_invariants(env, cur_env, idx);
+			if (log_level2)
+				log_env_changes(env, LOG_AT_TRANSFER, old_env, cur_env, idx);
+			/*
+			 * Treat nested loop exits as successors,
+			 * join cur_env into successor's envs.
+			 */
+			for (i = 0; i < nested_loop->exits_cnt; i++) {
+				s = nested_loop->exits[i].to;
+				err = join_successor(env, cur_header, s, old_env, cur_env);
+				if (err)
+					goto out;
+			}
+		} else {
+			/*
+			 * Iterate instructions within a single basic block
+			 * starting at 'idx' mutating 'cur_env'.
+			 */
+			for (;;) {
+				if (log_level2)
+					memcpy(old_env, cur_env, sizeof(*old_env));
+				err = transfer(env, cur_env, idx);
+				if (err)
+					goto out;
+				reset_scevs_at_indirect_writes(env, cur_env, idx);
+				if (log_level2)
+					log_env_changes(env, LOG_AT_TRANSFER, old_env, cur_env, idx);
+				succ = bpf_insn_successors(env, idx);
+				if (succ->cnt != 1)
+					break;
+				succ_idx = succ->items[0];
+				if (aux[idx].bb_end || aux[succ_idx].need_scev ||
+				    bpf_loop_at_index(env, succ_idx) != cur_header)
+					break;
+				idx = succ_idx;
+			}
+			/* Join cur_env into basic block successor's envs. */
+			iarray_for_each(s, succ) {
+				err = join_successor(env, cur_header, s, old_env, cur_env);
+				if (err)
+					goto out;
+			}
+		}
+	}
+
+	err = 0;
+out:
+	kfree(cur_env);
+	kfree(old_env);
+	return err;
+nomem:
+	err = -ENOMEM;
+	goto out;
+}
+
+static void mark_latches(struct bpf_verifier_env *env)
+{
+	struct bpf_insn_aux_data *aux = env->insn_aux_data;
+	struct bpf_loop *loop;
+	int len = env->prog->len;
+	int i, j, k, from;
+
+	for (i = 0; i < len; i++) {
+		loop = aux[i].loop;
+		if (!loop)
+			continue;
+		aux[i].need_scev = true;
+		if (!can_compute_loop_scev(loop))
+			continue;
+		/*
+		 * Make sure to only set .need_scev for exits and latches
+		 * belonging to the loop at `i`. If can_compute_loop_scev()
+		 * fails for a loop, it should hide it's instructions from
+		 * compute_insn_scevs().
+		 */
+		for (j = 0; j < loop->backedges_cnt; j++)
+			for (k = 0; k < loop->backedges[j].latches_cnt; k++)
+				aux[loop->backedges[j].latches[k]].need_scev = true;
+		for (j = 0; j < loop->exits_cnt; j++) {
+			from = loop->exits[j].from;
+			if (bpf_loop_at_index(env, from) == i)
+				aux[from].need_scev = true;
+		}
+	}
+}
+
+static bool is_any_imm_reg_opaque(struct scev *scev, u32 id)
+{
+	u32 l, r, reg, order;
+	s64 imm;
+
+	scev->stack_sz = 0;
+	expr_stack_push(scev, id);
+	while (expr_next(scev, &id, &order)) {
+		if (order & DEPTH_LIMIT)
+			return false;
+		if (!(order & PRE) ||
+		    is_imm(scev, id, &imm) ||
+		    is_reg(scev, id, &reg) ||
+		    is_opaque(scev, id) ||
+		    is_any(scev, id, &l, &r))
+			continue;
+		return false;
+	}
+	return true;
+}
+
+/*
+ * Can implement explicit stack version, but it is harder to read.
+ * Stick with recursive version for now.
+ */
+static int transform_expr_once(struct scev *scev, u32 lvl, u32 root, void *priv,
+			       int (*fn)(struct scev *scev, u32 id, void *priv))
+{
+	struct expr expr;
+	int p0, p1, id;
+
+	if (lvl >= EXPR_STACK_DEPTH)
+		return UNKNOWN_EXPR_ID;
+
+	expr = scev->exprs[root]; /* snapshot the expr before potential realloc */
+	switch (op_params_num(expr.op)) {
+	case 0:
+		id = root;
+		break;
+	case 1:
+		p0 = transform_expr_once(scev, lvl + 1, expr.params[0], priv, fn);
+		id = expr1(scev, expr.op, p0);
+		break;
+	case 2:
+		p0 = transform_expr_once(scev, lvl + 1, expr.params[0], priv, fn);
+		p1 = transform_expr_once(scev, lvl + 1, expr.params[1], priv, fn);
+		id = expr2(scev, expr.op, p0, p1);
+		break;
+	}
+	return id < 0 ? id : fn(scev, id, priv);
+}
+
+static int transform_expr(struct scev *scev, u32 root, void *priv,
+			  int (*fn)(struct scev *scev, u32 id, void *priv))
+{
+	int i = 0, id_old, id_new = root;
+
+	do {
+		/*
+		 * The depth limit is somewhat arbitrary and is essentially a failsafe.
+		 * oscillation should not really happen. Still, it seems harsh to error out.
+		 */
+		if (i++ == 4 * EXPR_STACK_DEPTH)
+			return UNKNOWN_EXPR_ID;
+		id_old = id_new;
+		id_new = transform_expr_once(scev, 0, id_old, priv, fn);
+		if (id_new < 0)
+			return id_new;
+	} while (id_old != id_new);
+	return id_new;
+}
+
+static int simplify(struct scev *scev, u32 id, void *priv)
+{
+	u32 l, r, base, slope;
+	s64 imm1, imm2;
+
+	/* (+ (linear base slope) imm) -> (linear (+ base imm) slope) */
+	if (is_add(scev, id, &l, &r) &&
+	    is_linear(scev, l, &base, &slope) &&
+	    is_imm(scev, r, &imm1))
+		return expr2(scev, LINEAR_SCEV, expr2(scev, BPF_ADD, base, r), slope);
+
+	/* (+ imm imm) -> imm */
+	if (is_add(scev, id, &l, &r) &&
+	    is_imm(scev, l, &imm1) &&
+	    is_imm(scev, r, &imm2))
+		return imm_expr(scev, imm1 + imm2);
+
+	/* (zext32 (imm X)) -> (imm (u32)X) */
+	if (is_zext32(scev, id, &l) && is_imm(scev, l, &imm1))
+		return imm_expr(scev, (u64)(u32)imm1);
+
+	return id;
+}
+
+static int compute_header_scevs(struct bpf_verifier_env *env, struct env *header_env)
+{
+	struct scev *scev = env->scev;
+	int l, r, id, ra, rb, ra_expr;
+	s64 imm;
+
+	/*
+	 * Note:
+	 * reg2expr[ra] describes a change in rA's value over a single loop iteration.
+	 * It does not account for rA's entry value. However, the SCEV constructed for rA
+	 * must include an additional rA entry edge. For example:
+	 *
+	 *      r0 = ..., r1 = 7
+	 *   l: if r0 == 10 goto e // reg2expr[r1] == (imm 42) => reg2scev[r1] == (any r1 42)
+	 *      r0 += 1
+	 *      r1 = 42
+	 *      goto l
+	 *   e:
+	 *      ... at exit r1 is in {7,42} ...
+	 */
+	for (ra = 0; ra < REGS_NUM; ra++) {
+		ra_expr = transform_expr(scev, header_env->reg2expr[ra], NULL, simplify);
+		if (ra_expr < 0)
+			return ra_expr;
+		header_env->reg2expr[ra] = ra_expr;
+		/* rA = (+ rA IMM) -> (linear rA IMM) */
+		if (is_add(scev, ra_expr, &l, &r) &&
+		    is_reg(scev, l, &rb) &&
+		    is_imm(scev, r, &imm) &&
+		    ra == rb) {
+			id = expr2(scev, LINEAR_SCEV, l, r);
+			if (id < 0)
+				return id;
+			header_env->reg2scev[ra] = id;
+			continue;
+		}
+		/* rA = rA -> rA*/
+		if (is_reg(scev, ra_expr, &rb) && ra == rb) {
+			header_env->reg2scev[ra] = ra_expr;
+			continue;
+		}
+		/* rA = (any 1 2 3 4 ...) -> (any rA 1 2 3 4 ...) */
+		if (is_any_imm_reg_opaque(scev, ra_expr)) {
+			id = mk_any(scev, expr1(scev, REG, ra), ra_expr);
+			if (id < 0)
+				return id;
+			header_env->reg2scev[ra] = id;
+			continue;
+		}
+
+	}
+	return 0;
+}
+static int instantiate_header_scevs(struct scev *scev, u32 id, void *priv)
+{
+	u32 reg, *reg2scev = priv;
+
+	/* (reg r) -> ... equation at header ... */
+	if (is_reg(scev, id, &reg))
+		return reg2scev[reg];
+	/* keep expression shapes */
+	return id;
+}
+
+static int compute_insn_scevs(struct bpf_verifier_env *env, struct env *eheader, struct env *einsn)
+{
+	struct scev *scev = env->scev;
+	int id, reg;
+
+	for (reg = 0; reg < REGS_NUM; reg++) {
+		id = einsn->reg2expr[reg];
+		id = transform_expr_once(scev, 0, id, eheader->reg2scev, instantiate_header_scevs);
+		if (id < 0)
+			return id;
+		id = transform_expr(scev, id, NULL, simplify);
+		if (id < 0)
+			return id;
+		einsn->reg2scev[reg] = id;
+	}
+
+	return 0;
+}
+
+static int log_scevs(struct bpf_verifier_env *env)
+{
+	struct bpf_insn_aux_data *aux = env->insn_aux_data;
+	struct bpf_verifier_log *log = &env->log;
+	struct env *header_env, *latch_env;
+	struct scev *scev = env->scev;
+	struct bpf_loop *loop;
+	int i, j, k, len, latch;
+
+	len = env->prog->len;
+	for (i = 0; i < len; i++) {
+		loop = aux[i].loop;
+		if (!loop)
+			continue;
+		header_env = find_header_env(scev, i);
+		bpf_log(log, "scev at header %d:\n", i);
+		print_env(env, header_env, i);
+		if (!can_compute_loop_scev(loop))
+			continue;
+		for (j = 0; j < loop->backedges_cnt; j++) {
+			for (k = 0; k < loop->backedges[j].latches_cnt; k++) {
+				latch = loop->backedges[j].latches[k];
+				bpf_log(log, " scev at %s %d:\n", k ? "exit" : "latch", latch);
+				latch_env = find_loop_env(scev, i, latch);
+				if (verifier_bug_if(!latch_env, env, "i=%d, latch=%d", i, latch))
+					return -EFAULT;
+				print_env(env, latch_env, latch);
+			}
+		}
+	}
+	return 0;
+}
+
+int bpf_compute_scev(struct bpf_verifier_env *env)
+{
+	struct bpf_insn_aux_data *aux = env->insn_aux_data;
+	struct scev *scev = env->scev;
+	int *postorder = env->cfg.insn_postorder;
+	int cnt = env->cfg.cur_postorder;
+	int i, idx, err, header;
+
+	mark_latches(env);
+	/*
+	 * Visit loop headers in postorder, to guarantee that scevs
+	 * for innermost loops are computed first.
+	 */
+	for (i = 0; i < cnt; i++) {
+		idx = postorder[i];
+		if (!aux[idx].loop)
+			continue;
+		err = compute_scev_for_loop(env, idx);
+		if (err)
+			return err;
+	}
+
+	/*
+	 * Compute scevs from exprs collected on a previous step. Iterate instructions in
+	 * reverse post-order so that each loop header is processed before instructions
+	 * reachable from it.
+	 */
+	for (i = cnt - 1; i >= 0; i--) {
+		idx = postorder[i];
+		if (!aux[idx].need_scev)
+			continue;
+		header = bpf_loop_at_index(env, idx);
+		err = aux[idx].loop
+		      ? compute_header_scevs(env, find_header_env(scev, header))
+		      : compute_insn_scevs(env,
+					   find_header_env(scev, header),
+					   find_loop_env(scev, header, idx));
+		if (err)
+			return err;
+	}
+
+	for (i = 0; i < env->prog->len; i++)
+		if (aux[i].loop)
+			aux[i].prune_point = true;
+
+	if (env->log.level & BPF_LOG_LEVEL2) {
+		err = log_scevs(env);
+		if (err)
+			return err;
+	}
+
+	return 0;
+}
+
+void bpf_free_scev(struct bpf_verifier_env *env)
+{
+	struct scev *scev = env->scev;
+	struct insn_envs *envs;
+	int i;
+	u32 j;
+
+	if (!scev)
+		return;
+	for (i = 0; i < scev->envs_cnt; i++) {
+		envs = scev->envs[i];
+		if (!envs)
+			continue;
+		for (j = 0; j < envs->cnt; j++)
+			kfree(envs->entries[j].env);
+		kfree(envs);
+	}
+	if (scev->exprs_ht)
+		for (i = 0; i < scev->exprs_ht_cnt; i++)
+			kvfree(scev->exprs_ht[i]);
+	kvfree(scev->exprs_ht);
+	kvfree(scev->worklist.data);
+	kvfree(scev->envs);
+	kvfree(scev->exprs);
+	kvfree(scev->discovered);
+	kfree(scev);
+	env->scev = NULL;
+}
+
+int bpf_init_scev(struct bpf_verifier_env *env)
+{
+	struct scev *scev;
+
+	scev = kzalloc(sizeof(struct scev), GFP_KERNEL_ACCOUNT);
+	if (!scev)
+		return -ENOMEM;
+	env->scev = scev;
+	/* A power of two for hash & (exprs_ht_cnt - 1)` indexing. */
+	scev->exprs_ht_cnt = roundup_pow_of_two(max(256U, DIV_ROUND_UP(env->prog->len, 4)));
+	scev->exprs_ht = kvcalloc(scev->exprs_ht_cnt, sizeof(*scev->exprs_ht), GFP_KERNEL_ACCOUNT);
+	if (!scev->exprs_ht)
+		goto nomem;
+	if (expr0(scev, UNKNOWN) < 0 || expr0(scev, OPAQUE)  < 0)
+		goto nomem;
+	scev->envs = kvcalloc(env->prog->len, sizeof(*scev->envs), GFP_KERNEL_ACCOUNT);
+	scev->discovered = kvcalloc(env->prog->len, sizeof(*scev->discovered), GFP_KERNEL_ACCOUNT);
+	if (!scev->envs || !scev->discovered)
+		goto nomem;
+	/*
+	 * Remember original program length, in case bpf_free_scev()
+	 * is called after bpf program rewrites that increase program
+	 * length.
+	 */
+	scev->envs_cnt = env->prog->len;
+	return 0;
+nomem:
+	bpf_free_scev(env);
+	return -ENOMEM;
+}
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 85971f9c09c4..60287049cfe4 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -23233,6 +23233,14 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (env->log.level & BPF_LOG_LEVEL2)
 		log_program(env);
 
+	ret = bpf_init_scev(env);
+	if (ret < 0)
+		goto skip_full_check;
+
+	ret = bpf_compute_scev(env);
+	if (ret < 0)
+		goto skip_full_check;
+
 	ret = mark_fastcall_patterns(env);
 	if (ret < 0)
 		goto skip_full_check;
@@ -23382,6 +23390,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 		bpf_clear_insn_aux_data(env, 0, env->insn_aux_data_len);
 	vfree(env->insn_aux_data);
 	kvfree(env->fd_array);
+	bpf_free_scev(env);
 	bpf_stack_liveness_free(env);
 	kvfree(env->cfg.postorder_nums);
 	kvfree(env->cfg.insn_postorder);

-- 
2.53.0

Re: [PATCH bpf-next v3 33/51] bpf: compute scalar evolution expressions for loops

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 2604 bytes --]

> bpf: compute scalar evolution expressions for loops

TL;DR: The new need_scev bool in struct bpf_insn_aux_data may grow it from 208
to 216 bytes, about 8MB more for a 1M-insn program; a bitfield would avoid
that. Also lacks a comment.

> diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
> index 963106328791..52f72e92eb0c 100644
> --- a/include/linux/bpf_verifier.h
> +++ b/include/linux/bpf_verifier.h

[ ... ]

> @@ -745,6 +745,7 @@ struct bpf_insn_aux_data {
>  	bool call_with_percpu_alloc_ptr; /* {this,per}_cpu_ptr() with prog percpu alloc */
>  	bool arena_scalar; /* ldx/stx/st/atomic through a number, it's an address in arena */
>  	bool bb_end; /* last instruction of a basic block */
> +	bool need_scev;

Does this new bool grow struct bpf_insn_aux_data?

Before this patch, the ten bools after 'u32 seen', plus 'u8 alu_state' and
the u8 fastcall bitfield byte, filled exactly 12 bytes, so 'unsigned int
orig_idx' followed them with no padding.  With one more bool, orig_idx and
every field after it move forward by 4 bytes, and because the trailing
'struct bpf_loop *loop' needs 8-byte alignment, the struct goes from 208 to
216 bytes on x86_64 (orig_idx moves from offset 136 to 140).

bpf_check() allocates this struct once per instruction:

    __vmalloc(array_size(sizeof(struct bpf_insn_aux_data), len))

and adjust_insn_aux_data() memmoves it on every patch, so a 1M instruction
program uses about 8MB more memory.  Nothing later in the series
(71f4568a6476..726d0559f576) touches the struct again, so the growth
remains.

There is room that would cost nothing: a 2-byte hole after 'u16
stack_ptrs', and 25 unused bits in the 'u32 jmp_point:1 ... jump_target:1'
word.  That word holds the other flags set once in the pre-pass, such as
prune_point, and need_scev is also set only once, by mark_latches().  Would
'u32 need_scev:1;' there keep the struct at its current size?

This isn't a bug, but this is also the only member in this block without a
comment.  Could one be added saying that the flag marks loop headers,
latches and in-loop exit sources where SCEVs are materialized, and that it
also forces a basic-block split in compute_scev_for_loop()?

>  	u8 alu_state; /* used in combination with alu_limit */
>  	/* true if STX or LDX instruction is a part of a spill/fill
>  	 * pattern for a bpf_fastcall call.


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/37923187511

[PATCH bpf-next v3 34/51] bpf: use SCEV to widen bounded loops

[Eduard Zingerman]

Loop-bound computation and register widening
===========================================

During the main verification pass, use SCEV expressions and register
values at the entry to the loop to estimate the number of iterations
the loop might execute. Use this number to widen loop induction variables.

For example:

    0: r7 = 5;
    1: r6 = 0;
    2: while (r6 < 3) {   // latch
    3:     if (r6 == 1)
    4:         r7 = 10;
    5:     r6 += 1;
    6: }
    8: use(r7);

- Scalar evolution expressions (SCEVs) at the latch are:
  - r6 = (linear r6 1)
  - r7 = (any r7 10)
- For this pre-condition loop, this corresponds to the linear inequality
  `r6_entry + n < 3`. With r6_entry = 0, the inequality is `n < 3`.
- Hence infer that r6's range at line 2 is [0,3], including the final
  failed test. The body starts with r6 in [0,2].
- r7's (any r7 10) expression means that its initial value 5 from line 0
  is joined with 10 from line 4: it is widened to [5,10] within
  and after the loop.

Currently such inference is supported for:
- Reducible loops with one backedge and complete backedge/exit lists.
- A linear latch dominating the backedge, with one successor exiting
  the loop. Additional exits are allowed; they can shorten execution.
- 64-bit arithmetic and comparisons whose continuation condition
  normalizes to signed/unsigned < or <=, or !=.
- Initial counter, latch offset, slope and bound evaluable as constants
  at loop entry, yielding a finite, representable iteration bound.
- Values with an eligible type at loop entry, e.g. SCALAR_VALUE or
  PTR_TO_STACK, but not PTR_TO_CTX.

In order for widening to proceed, every register live at the loop
entry has to have a SCEV that can be widened. Currently, the following
expressions are supported:
- (linear <self> <step>): entry value + iteration# * constant step.
- (reg <self>): loop invariant; keep its entry value unchanged.
- (any ...): join the entry value with immediates or invariant registers;
  types must match, and immediates require SCALAR_VALUE.

Main verification loop integration
==================================

Before proceeding with the usual instruction processing, do_check()
takes the following steps at loop entries (H):
- Checks whether the loop can be widened and obtains its
  iteration bound (scev.c:bpf_compute_loop_iters()).
- If it can, saves an unwidened checkpoint E0, which will be used as
  a base for deriving widened values throughout this invocation of
  the loop.
- Widens registers in the current state using E0 and SCEV expressions
  computed for H (scev.c:bpf_widen_scev_regs()).
- Pushes a per-frame loop-stack record to the current verifier state.
  For terminating loops, the record stores E0 and the iteration bounds.
- Lets is_state_visited() create a new checkpoint W at the same
  header, now containing the widened values, and proceeds with
  normal checking.
- On an exit edge, pops the loop record and verifies the exit
  path normally.

is_state_visited() treats a loop proven to terminate like an
iterator-based loop: RANGE_WITHIN comparison attempts to establish
convergence against a checkpoint whose exploration is still
in progress.

For example:

    r0 = 0;
    H: r0 += 1;
       if (r0 != 3) goto H;
       exit;

    first H:  prove H_count = 3; save E0(r0 = 0)
              widen r0 to [0,2]; push the loop record
              save W(r0 = [0,2])
    body:     r0 becomes [1,3]; fork at the conditional
    exit:     r0 = 3; pop the loop record and check exit
    backedge: r0 = [1,2]; clamp, then match with W and prune

Clamping
--------

Independent register ranges lose correlations between induction
variables. For example:

    r6 = 0; r7 = 0;
    do { r6 += 1; r7 += 2; } while (r6 < 3);
    use(r7);

    header:   r6 in [0,2], r7 in {0,2,4}
    body:     r6 in [1,3], r7 in {2,4,6}
    backedge: r6 in [1,2], r7 still in {2,4,6}

The latch narrows r6 but does not propagate this refinement to r7.
To account for this, before comparing states on a backedge,
scev.c:bpf_clamp_scev_regs() recomputes r7's range from E0 and the
saved iteration bound, then intersects the current range {2,4,6}
with the recomputed {0,2,4}, arriving at {2,4}.

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 include/linux/bpf_verifier.h                       |   33 +
 kernel/bpf/log.c                                   |    5 +
 kernel/bpf/scev.c                                  | 1039 ++++++++++++++++++++
 kernel/bpf/states.c                                |   88 +-
 kernel/bpf/verifier.c                              |  205 +++-
 tools/testing/selftests/bpf/progs/verifier_gotox.c |    6 +-
 6 files changed, 1364 insertions(+), 12 deletions(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 52f72e92eb0c..d760d2283ac5 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -328,6 +328,25 @@ struct bpf_retval_range {
 	bool return_32bit;
 };
 
+/* SCEV/widening register space: r0..r10 plus every stack slot of a frame. */
+#define BPF_SCEV_REGS_NUM (MAX_BPF_REG + MAX_BPF_STACK_SLOTS)
+
+struct bpf_loop_iters {
+	u32 min_header_count;   /* min number of times header is executed */
+	u32 max_header_count;   /* max number of times header is executed */
+	unsigned long precise_regs[BITS_TO_LONGS(BPF_SCEV_REGS_NUM)];
+};
+
+struct loop_stack_entry {
+	struct bpf_verifier_state *entry_state;
+	u32 min_header_count;
+	u32 max_header_count;
+	u32 loop_id:31;
+	u32 terminates:1;
+};
+
+#define LOOP_STACK_SIZE 16
+
 /* state of the program:
  * type of all registers and stack info
  */
@@ -373,6 +392,11 @@ struct bpf_func_state {
 	u32 callback_depth;
 	/* Instructions processed in this frame and callees on the current path. */
 	u32 insns_subtotal;
+	/*
+	 * Control-flow loop nesting at the current insn within this frame's
+	 * subprogram (loops never cross subprogram boundaries).
+	 */
+	u32 loop_stack_cnt;
 
 	/* The following fields should be last. See copy_func_state() */
 	/* The state of the stack. Each element of the array describes BPF_REG_SIZE
@@ -390,6 +414,7 @@ struct bpf_func_state {
 
 	u16 out_stack_arg_cnt; /* Number of outgoing on-stack argument slots */
 	struct bpf_reg_state *stack_arg_regs; /* Outgoing on-stack arguments */
+	struct loop_stack_entry *loop_stack;
 };
 
 #define MAX_CALL_FRAMES 16
@@ -436,6 +461,7 @@ static_assert(MAX_BPF_STACK_SLOTS <= (1 << 12));
 #define MAX_STACK_ARG_SLOTS (MAX_BPF_FUNC_ARGS - MAX_BPF_FUNC_REG_ARGS)
 #define BPF_ID_MAP_SIZE ((MAX_BPF_REG + MAX_BPF_STACK_SLOTS + MAX_STACK_ARG_SLOTS) * \
 			 MAX_CALL_FRAMES)
+
 struct bpf_verifier_state {
 	/* call stack tracking */
 	struct bpf_func_state *frame[MAX_CALL_FRAMES];
@@ -1971,4 +1997,11 @@ int bpf_init_scev(struct bpf_verifier_env *env);
 void bpf_free_scev(struct bpf_verifier_env *env);
 int bpf_compute_scev(struct bpf_verifier_env *env);
 
+int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
+			   struct bpf_loop_iters *iters);
+int bpf_widen_scev_regs(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
+			struct bpf_verifier_state *loop_entry, struct bpf_loop_iters *iters);
+int bpf_clamp_scev_regs(struct bpf_verifier_env *env, struct bpf_func_state *st, u32 insn_idx,
+			struct bpf_verifier_state *entry_state, u32 max_header_count);
+
 #endif /* _LINUX_BPF_VERIFIER_H */
diff --git a/kernel/bpf/log.c b/kernel/bpf/log.c
index 8ee7d7b71147..5432c0fc442b 100644
--- a/kernel/bpf/log.c
+++ b/kernel/bpf/log.c
@@ -792,6 +792,11 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifie
 		verbose(env, " cb");
 	if (state->in_async_callback_fn)
 		verbose(env, " async_cb");
+	if (state->loop_stack_cnt) {
+		verbose(env, " loop_stack=");
+		for (i = 0; i < state->loop_stack_cnt; i++)
+			verbose(env, "%s%d", i ? "," : "", state->loop_stack[i].loop_id);
+	}
 	verbose(env, "\n");
 	if (!print_all)
 		mark_verifier_state_clean(env);
diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
index 2d2a84156180..05dd8e439609 100644
--- a/kernel/bpf/scev.c
+++ b/kernel/bpf/scev.c
@@ -1,11 +1,14 @@
 // SPDX-License-Identifier: GPL-2.0-only
 /* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */
 
+#include "linux/cnum.h"
 #include <linux/bpf_verifier.h>
 #include <linux/jhash.h>
 #include <linux/log2.h>
 #include <linux/min_heap.h>
 #include <linux/bug.h>
+#include <linux/tnum.h>
+#include <linux/overflow.h>
 
 #define REGS_NUM (MAX_BPF_REG + MAX_BPF_STACK_SLOTS)
 #define UNKNOWN_EXPR_ID 0
@@ -80,6 +83,9 @@ struct expr_stack_elt {
 DEFINE_MIN_HEAP(int, scev_worklist);
 
 struct scev {
+	/* Read-only representatives for stack slots without a saved register. */
+	struct bpf_reg_state zero_reg;
+	struct bpf_reg_state unknown_reg;
 	/*
 	 * Expressions are identified by id, exprs_ht ensures that
 	 * each expression exists as a unique instance.
@@ -1616,6 +1622,8 @@ int bpf_init_scev(struct bpf_verifier_env *env)
 	if (!scev)
 		return -ENOMEM;
 	env->scev = scev;
+	bpf_mark_reg_known_scalar(&scev->zero_reg, 0);
+	bpf_mark_reg_unknown_imprecise(&scev->unknown_reg);
 	/* A power of two for hash & (exprs_ht_cnt - 1)` indexing. */
 	scev->exprs_ht_cnt = roundup_pow_of_two(max(256U, DIV_ROUND_UP(env->prog->len, 4)));
 	scev->exprs_ht = kvcalloc(scev->exprs_ht_cnt, sizeof(*scev->exprs_ht), GFP_KERNEL_ACCOUNT);
@@ -1638,3 +1646,1034 @@ int bpf_init_scev(struct bpf_verifier_env *env)
 	bpf_free_scev(env);
 	return -ENOMEM;
 }
+
+static struct bpf_stack_state *scev_regno_to_stack(const struct bpf_func_state *st, u32 r)
+{
+	int spi, slots_available;
+
+	if (r < MAX_BPF_REG)
+		return NULL;
+
+	slots_available = st->allocated_stack / BPF_REG_SIZE;
+	spi = r - MAX_BPF_REG;
+	if (spi < slots_available)
+		return &st->stack[spi];
+
+	return NULL;
+}
+
+static struct bpf_reg_state *scev_regno_to_reg(struct bpf_func_state *st, u32 r)
+{
+	return r < MAX_BPF_REG ? &st->regs[r] : NULL;
+}
+
+/* Linear widening requires a physical register or a full spill. */
+static struct bpf_reg_state *scev_reg_for_linear(struct bpf_func_state *st, u32 r)
+{
+	struct bpf_stack_state *stack;
+
+	if (r < MAX_BPF_REG)
+		return scev_regno_to_reg(st, r);
+	stack = scev_regno_to_stack(st, r);
+	if (!stack || bpf_spill_size(stack) != BPF_REG_SIZE)
+		return NULL;
+	return &stack->spilled_ptr;
+}
+
+/*
+ * Return a borrowed value; never use it as a widening destination.
+ * NULL denotes no readable value.
+ */
+static const struct bpf_reg_state *
+scev_reg_for_read(struct bpf_verifier_env *env, const struct bpf_func_state *st, u32 r)
+{
+	const struct bpf_stack_state *stack;
+	bool zero = true;
+	int i;
+
+	if (r < MAX_BPF_REG)
+		return st->regs[r].type == NOT_INIT ? NULL : &st->regs[r];
+	if (r >= REGS_NUM || r - MAX_BPF_REG >= env->stack_limit / BPF_REG_SIZE)
+		return NULL;
+
+	stack = scev_regno_to_stack(st, r);
+	if (!stack)
+		return &env->scev->unknown_reg;
+	if (bpf_spill_size(stack) == BPF_REG_SIZE)
+		return &stack->spilled_ptr;
+
+	/* Partial spills do not describe the value of the whole slot. */
+	for (i = 0; i < BPF_REG_SIZE; i++) {
+		switch (stack->slot_type[i]) {
+		case STACK_ZERO:
+			break;
+		case STACK_INVALID:
+		case STACK_POISON:
+		case STACK_MISC:
+			zero = false;
+			break;
+		case STACK_SPILL:
+			if (!bpf_is_spilled_scalar_reg(stack))
+				return NULL;
+			zero = false;
+			break;
+		default:
+			return NULL;
+		}
+	}
+	return zero ? &env->scev->zero_reg : &env->scev->unknown_reg;
+}
+
+/* Materialize a writable value in the current state, growing its stack if needed. */
+static int scev_reg_for_write(struct bpf_verifier_env *env, struct bpf_func_state *st,
+			      u32 r, struct bpf_reg_state **out)
+{
+	const struct bpf_reg_state *value;
+	struct bpf_stack_state *stack;
+	u32 spi;
+	int err;
+
+	*out = scev_regno_to_reg(st, r);
+	if (*out)
+		return 0;
+	if (r >= REGS_NUM || r - MAX_BPF_REG >= env->stack_limit / BPF_REG_SIZE)
+		return -EINVAL;
+
+	spi = r - MAX_BPF_REG;
+	err = bpf_grow_stack_state(env, st, (spi + 1) * BPF_REG_SIZE);
+	if (err)
+		return err;
+
+	/* Resolve pointers after growth, which may relocate the stack. */
+	value = scev_reg_for_read(env, st, r);
+	if (!value)
+		return -EINVAL;
+	stack = scev_regno_to_stack(st, r);
+	stack->spilled_ptr = *value;
+	memset(stack->slot_type, STACK_SPILL, sizeof(stack->slot_type));
+	*out = &stack->spilled_ptr;
+	return 0;
+}
+
+static bool scev_reg_alive(struct bpf_verifier_env *env, struct bpf_verifier_state *st, u32 r)
+{
+	int insn_idx = bpf_frame_insn_idx(st, st->curframe);
+	u16 live_regs = env->insn_aux_data[insn_idx].live_regs_before;
+	int spi;
+
+	if (r < MAX_BPF_REG) {
+		return BIT(r) & live_regs;
+	} else {
+		spi = r - MAX_BPF_REG;
+		return bpf_stack_slot_alive(env, st->curframe, spi * 2) ||
+		       bpf_stack_slot_alive(env, st->curframe, spi * 2 + 1);
+	}
+}
+
+/*
+ * Latch is a condition deciding if execution remains inside a loop.
+ * Linear latch represents a condition 'if <reg> <op> <loop invariant> goto <loop-header>',
+ * where equation '<base> + i * <step> <op> <bound>' describes values taken by register <reg>,
+ * 'i' is the loop iteration number, starting from 0.
+ */
+struct linear_latch {
+	u32 insn_idx;
+	u32 base_expr;
+	u32 step_expr;
+	u32 bound_expr;
+	u32 op;
+};
+
+static bool loop_invariant(struct scev *scev, u32 id)
+{
+	s64 imm;
+	u32 reg;
+
+	return is_reg(scev, id, &reg) || is_imm(scev, id, &imm);
+}
+
+static int match_linear_latch(struct bpf_verifier_env *env,
+			      u32 header,
+			      u32 latch_idx,
+			      struct linear_latch *latch)
+{
+	struct bpf_insn *insn = &env->prog->insnsi[latch_idx];
+	struct scev *scev = env->scev;
+	struct env *latch_env;
+	u32 true_branch_loop;
+	u32 true_branch_tgt;
+	u32 src_reg_scev;
+	u32 dst_reg_scev;
+	u32 t, l, r, op;
+	int id;
+
+	/* 32-bit arithmetic is not handled yet */
+	if (BPF_CLASS(insn->code) != BPF_JMP)
+		return false;
+	op = BPF_OP(insn->code);
+	/* Flip the condition if true branch jumps out of the loop. */
+	true_branch_tgt = latch_idx + bpf_jmp_offset(insn) + 1;
+	true_branch_loop = bpf_loop_at_index(env, true_branch_tgt);
+	if (true_branch_loop != header &&
+	    !bpf_is_nested_loop(env, true_branch_loop, header))
+		op = bpf_rev_opcode(op);
+	switch (op) {
+	case BPF_JSLT:
+	case BPF_JSLE:
+	case BPF_JSGT:
+	case BPF_JSGE:
+	case BPF_JLT:
+	case BPF_JLE:
+	case BPF_JGT:
+	case BPF_JGE:
+	case BPF_JNE:
+		break;
+	default:
+		return false;
+	}
+
+	latch->insn_idx = latch_idx;
+
+	latch_env = find_loop_env(scev, header, latch_idx);
+	if (verifier_bug_if(!latch_env, env, "header_idx=%d, latch_idx=%d\n", header, latch_idx))
+		return -EFAULT;
+	dst_reg_scev = latch_env->reg2scev[insn->dst_reg];
+	src_reg_scev = latch_env->reg2scev[insn->src_reg];
+	/*
+	 * Make sure (linear ...) is in dst_reg_scev:
+	 *   'if <invariant> <op> <linear>' ⇔
+	 * 'if <linear> <flip(op)> <invariant>'.
+	 */
+	if (BPF_SRC(insn->code) == BPF_X && is_linear(scev, src_reg_scev, &l, &r)) {
+		swap(dst_reg_scev, src_reg_scev);
+		op = bpf_flip_opcode(op);
+	}
+	latch->op = op;
+	/*
+	 * simplify() produces two shapes for linear latches:
+	 * - (linear register step)
+	 *           ^----------------------------+-- base_expr
+	 *           base_expr                    |
+	 * - (linear (+ register imm) step)       |
+	 *           ^----------------------------'
+	 */
+	if (!is_linear(scev, dst_reg_scev, &latch->base_expr, &latch->step_expr))
+		return false;
+
+	if (BPF_SRC(insn->code) == BPF_K) {
+		id = imm_expr(scev, insn->imm);
+		if (id < 0)
+			return id;
+		latch->bound_expr = id;
+	} else {
+		latch->bound_expr = src_reg_scev;
+	}
+
+	if (!loop_invariant(scev, latch->step_expr) ||
+	    !loop_invariant(scev, latch->bound_expr))
+		return false;
+
+	/* (linear register step) */
+	if (is_reg(scev, latch->base_expr, &t))
+		return true;
+
+	/* (linear (+ register imm) step) */
+	if (is_add(scev, latch->base_expr, &l, &r) &&
+	    is_reg(scev, l, &t) &&
+	    loop_invariant(scev, r))
+		return true;
+
+	return false;
+}
+
+struct scev_value {
+	u64 value;	/* scalar value or a pointer offset */
+	int ptr_reg;	/* SCEV register identifying the pointer origin, or -1. */
+};
+
+static void log_scev_value(struct bpf_verifier_env *env, struct bpf_func_state *st,
+			   const struct scev_value *v)
+{
+	const struct bpf_reg_state *reg;
+
+	if (v->ptr_reg >= 0) {
+		reg = scev_reg_for_read(env, st, v->ptr_reg);
+		log_reg(env, v->ptr_reg);
+		bpf_log(&env->log, " (%s)", reg ? reg_type_str(env, reg->type) : "unavailable");
+	} else {
+		bpf_log(&env->log, "scalar %llu", v->value);
+	}
+}
+
+static void log_incompatible_scev_values(struct bpf_verifier_env *env, const char *reason,
+					struct bpf_func_state *st,
+					const struct scev_value *a, const struct scev_value *b)
+{
+	bpf_log(&env->log, "scev: %s ", reason);
+	log_scev_value(env, st, a);
+	bpf_log(&env->log, ", ");
+	log_scev_value(env, st, b);
+	bpf_log(&env->log, "\n");
+}
+
+static bool scev_value_compatible(struct bpf_verifier_env *env, struct bpf_func_state *st,
+				  const struct scev_value *a, const struct scev_value *b)
+{
+	const struct bpf_reg_state *ra, *rb;
+
+	if (a->ptr_reg < 0 || b->ptr_reg < 0) {
+		if (a->ptr_reg < 0 && b->ptr_reg < 0)
+			return true;
+		goto incompatible;
+	}
+	ra = scev_reg_for_read(env, st, a->ptr_reg);
+	rb = scev_reg_for_read(env, st, b->ptr_reg);
+	if (ra && rb && bpf_same_memory_origin(ra, rb))
+		return true;
+
+incompatible:
+	if (env->log.level & BPF_LOG_LEVEL2)
+		log_incompatible_scev_values(env, "incompatible latch operands", st, a, b);
+	return false;
+}
+
+static bool scev_value_add(struct bpf_verifier_env *env, struct bpf_func_state *st,
+			   const struct scev_value *a, const struct scev_value *b,
+			   struct scev_value *result)
+{
+	if (a->ptr_reg >= 0 && b->ptr_reg >= 0) {
+		if (env->log.level & BPF_LOG_LEVEL2)
+			log_incompatible_scev_values(env, "can't add two pointers", st, a, b);
+		return false;
+	}
+	result->value = a->value + b->value;
+	result->ptr_reg = a->ptr_reg >= 0 ? a->ptr_reg : b->ptr_reg;
+	return true;
+}
+
+static bool eval_expr(struct bpf_verifier_env *env, struct scev *scev,
+		      struct bpf_func_state *st, u32 id, struct scev_value *result)
+{
+	struct scev_value lval, rval;
+	const struct bpf_reg_state *reg;
+	u32 l, r, regno;
+	s64 imm;
+
+	if (is_reg(scev, id, &regno)) {
+		reg = scev_reg_for_read(env, st, regno);
+		if (!reg || reg->type == NOT_INIT || !tnum_is_const(reg->var_off))
+			return false;
+		result->value = reg->var_off.value;
+		result->ptr_reg = reg->type == SCALAR_VALUE ? -1 : (int)regno;
+		return true;
+	} else if (is_imm(scev, id, &imm)) {
+		result->value = imm;
+		result->ptr_reg = -1;
+		return true;
+	} else if (is_add(scev, id, &l, &r) &&
+		   eval_expr(env, scev, st, l, &lval) &&
+		   eval_expr(env, scev, st, r, &rval)) {
+		return scev_value_add(env, st, &lval, &rval, result);
+	}
+	return false;
+}
+
+/*
+ * Loop with post-condition:
+ *
+ *    r0 = 0
+ * l: ...                r0 ∈ [0,1,2] header executed 3 times
+ *    r0 += 1            r0 ∈ [0,1,2]
+ *    ...                r0 ∈ [1,2,3]
+ *    if r0 != 3 goto l  r0 ∈ [1,2,3] backedge taken 2 times
+ *    ...                r0 ∈ [3]
+ *
+ * SCEV at header: r0 = k
+ * SCEV at latch:  r0 = 1 + k
+ *
+ * Loop with pre-condition:
+ *
+ *    r0 = 0
+ * l: ...                r0 ∈ [0,1,2,3] header executed 4 times
+ *    if r0 == 3 goto e  r0 ∈ [0,1,2,3] backedge taken 3 times
+ *    r0 += 1            r0 ∈ [0,1,2]
+ *    ...                r0 ∈ [1,2,3]
+ *    goto l             r0 ∈ [1,2,3]
+ * e: ...                r0 ∈ [3]
+ *
+ * SCEV at header: r0 = k
+ * SCEV at latch:  r0 = k
+ */
+static bool compute_max_iters(struct bpf_verifier_env *env,
+			      struct bpf_func_state *st,
+			      struct linear_latch *latch,
+			      struct bpf_loop_iters *iters)
+{
+	struct bpf_insn_aux_data *aux = env->insn_aux_data;
+	struct bpf_loop *loop = aux[bpf_loop_at_index(env, latch->insn_idx)].loop;
+	struct scev *scev = env->scev;
+	struct scev_value initial_val, step_val, bound_val;
+	u64 step, diff, bound, initial, backedge_taken_count;
+	u8 op = latch->op;
+	bool up, down;
+
+	if (!eval_expr(env, scev, st, latch->base_expr, &initial_val) ||
+	    !eval_expr(env, scev, st, latch->step_expr, &step_val) ||
+	    !eval_expr(env, scev, st, latch->bound_expr, &bound_val))
+		return false;
+
+	if (step_val.ptr_reg >= 0) {
+		if (env->log.level & BPF_LOG_LEVEL2) {
+			bpf_log(&env->log, "scev: latch step is a pointer ");
+			log_scev_value(env, st, &step_val);
+			bpf_log(&env->log, "\n");
+		}
+		return false;
+	}
+	if (!scev_value_compatible(env, st, &initial_val, &bound_val))
+		return false;
+
+	initial = initial_val.value;
+	step = step_val.value;
+	bound = bound_val.value;
+
+	if (step == 0)
+		return false;
+	down = (s64)step < 0;
+	up = (s64)step > 0;
+	diff = bound - initial;
+	switch (op) {
+	case BPF_JLT:
+		/*
+		 * E.g.: 3 + 2*i < 7 ⇔ 2*i < 4 ⇔ i ∈ [0,1].
+		 * Here the inequality describes a set of values `i` might take
+		 * while the latch condition remains true (the backedge is taken).
+		 * The backedge_taken_count is the size of this set.
+		 */
+		if (/*
+		     * If the condition is initially true, decreasing values won't
+		     * change that without underflow. If the condition is initially false,
+		     * the main pass would do a single iteration anyway.
+		     */
+		    down ||
+		    /* Is the latch condition false on the first iteration? */
+		    initial >= bound ||
+		    /* Count iterations satisfying the strict inequality, including i=0. */
+		    check_add_overflow((diff - 1) / step, 1, &backedge_taken_count) ||
+		    /*
+		     * Check if the last iteration overflows the counter:
+		     *   initial + backedge_taken_count * step > U64_MAX
+		     */
+		    backedge_taken_count > (U64_MAX - initial) / step)
+			return false;
+		break;
+	case BPF_JLE:
+		/* E.g.: 3 + 2*i ≤ 7 ⇔ 2*i ≤ 4 ⇔ i ∈ [0,1,2]. */
+		if (down || initial > bound ||
+		    /* Count iterations satisfying the non-strict inequality. */
+		    check_add_overflow(diff / step, 1, &backedge_taken_count) ||
+		    backedge_taken_count > (U64_MAX - initial) / step)
+			return false;
+		break;
+	case BPF_JGT:
+		/* E.g.: 7 - 2*i > 3 ⇔ 2*i < 4 ⇔ i ∈ [0,1]. */
+		if (up || initial <= bound ||
+		    check_add_overflow((-diff - 1) / -step, 1, &backedge_taken_count) ||
+		    /*
+		     * The last iteration underflows the counter if:
+		     *   initial + backedge_taken_count * step < 0 ⇔
+		     *             backedge_taken_count * step < -initial ⇔
+		     *                    backedge_taken_count > initial / -step
+		     * (direction flips because step < 0)
+		     */
+		    backedge_taken_count > initial / -step)
+			return false;
+		break;
+	case BPF_JGE:
+		/* E.g.: 7 - 2*i ≥ 3 ⇔ 2*i ≤ 4 ⇔ i ∈ [0,1,2]. */
+		if (up || initial < bound ||
+		    check_add_overflow(-diff / -step, 1, &backedge_taken_count) ||
+		    backedge_taken_count > initial / -step)
+			return false;
+		break;
+	case BPF_JSLT:
+		if (down || (s64)initial >= (s64)bound ||
+		    check_add_overflow((diff - 1) / step, 1, &backedge_taken_count) ||
+		    /*
+		     * Check if the last iteration overflows the signed counter:
+		     *   (s64)initial + backedge_taken_count * step > S64_MAX
+		     */
+		    backedge_taken_count > ((u64)S64_MAX - initial) / step)
+			return false;
+		break;
+	case BPF_JSLE:
+		if (down || (s64)initial > (s64)bound ||
+		    check_add_overflow(diff / step, 1, &backedge_taken_count) ||
+		    backedge_taken_count > ((u64)S64_MAX - initial) / step)
+			return false;
+		break;
+	case BPF_JSGT:
+		if (up || (s64)initial <= (s64)bound ||
+		    check_add_overflow((-diff - 1) / -step, 1, &backedge_taken_count) ||
+		    /*
+		     * The last iteration underflows the signed counter if:
+		     *   (s64)initial + backedge_taken_count * step < S64_MIN ⇔
+		     *                  backedge_taken_count * step < S64_MIN - (s64)initial ⇔
+		     *                         backedge_taken_count > initial - (u64)S64_MIN / -step
+		     *	 (direction flips because step < 0)
+		     */
+		    backedge_taken_count > (initial - (u64)S64_MIN) / -step)
+			return false;
+		break;
+	case BPF_JSGE:
+		if (up || (s64)initial < (s64)bound ||
+		    check_add_overflow(-diff / -step, 1, &backedge_taken_count) ||
+		    backedge_taken_count > (initial - (u64)S64_MIN) / -step)
+			return false;
+		break;
+	case BPF_JNE:
+		/*
+		 * For a decreasing counter flip the diff and step:
+		 *   7 - 2*i ≠ 3 ⇔ 2*i ≠ 7 - 3.
+		 */
+		diff = down ? -diff : diff;
+		step = down ? -step : step;
+		if (diff == 0 || diff % step)
+			return false;
+		backedge_taken_count = diff / step;
+		break;
+	default:
+		return false;
+	}
+
+	/*
+	 * max_header_count is u32 and U32_MAX means "infinite",
+	 * check to avoid overflow below.
+	 */
+	if (backedge_taken_count >= U32_MAX - 1)
+		return false;
+
+	/* The header executes once on entry and once for each taken backedge, hence +1. */
+	iters->max_header_count = backedge_taken_count + 1;
+	/*
+	 * The latch is a conditional jump with one jump target exiting the loop.
+	 * Linear latch is matched only if the loop has a single backedge.
+	 * The loop still, however can have multiple exits.
+	 * In such case, conservatively assume that non-latch exit can happen
+	 * at any iteration, thus setting minimal number of iterations as 0.
+	 */
+	iters->min_header_count = loop->exits_cnt == 1 ? iters->max_header_count : 0;
+	return true;
+}
+
+static void mark_scev_reg_scratched(struct bpf_verifier_env *env, u32 r)
+{
+	if (r < MAX_BPF_REG)
+		mark_reg_scratched(env, r);
+	else
+		mark_stack_slot_scratched(env, r - __MAX_BPF_REG);
+}
+
+/*
+ * An unknown scalar is its own widening.
+ * Stack bytes without a value are represented by scev->unknown_reg.
+ */
+static bool is_unbound_scev_reg(struct bpf_verifier_env *env, const struct bpf_func_state *st,
+				u32 r)
+{
+	const struct bpf_reg_state *reg = scev_reg_for_read(env, st, r);
+
+	return reg && reg->type == SCALAR_VALUE && !reg->id && reg->step <= 1 &&
+	       tnum_is_unknown(reg->var_off) &&
+	       reg->r64.size == U64_MAX && reg->r32.size == U32_MAX;
+}
+
+/* Main logic in verifier.c forbids varying offsets for certain register types. */
+static bool is_widenable_reg_type(const struct bpf_reg_state *reg)
+{
+	if (type_may_be_null(reg->type))
+		return false;
+
+	switch (base_type(reg->type)) {
+	case SCALAR_VALUE:
+	case PTR_TO_MAP_VALUE:
+	case PTR_TO_MAP_KEY:
+	case PTR_TO_STACK:
+	case PTR_TO_PACKET:
+	case PTR_TO_PACKET_META:
+	case PTR_TO_MEM:
+	case PTR_TO_BUF:
+	case PTR_TO_BTF_ID:
+		return true;
+	default:
+		return false;
+	}
+}
+
+static void scratch_widened_reg_id(struct bpf_verifier_env *env, struct bpf_reg_state *reg)
+{
+	switch (base_type(reg->type)) {
+	case SCALAR_VALUE:
+		reg->id = 0;
+		break;
+	case PTR_TO_PACKET:
+	case PTR_TO_PACKET_META:
+		reg->id = ++env->id_gen;
+		break;
+	default:
+		break;
+	}
+}
+
+/* Check that ANY leaves can be unioned with r's loop-entry value. */
+static bool is_widenable_any(struct bpf_verifier_env *env, struct bpf_func_state *loop_entry,
+			     struct env *header_env, u32 r, u32 id)
+{
+	const struct bpf_reg_state *reg, *leaf_reg;
+	struct scev *scev = env->scev;
+	u32 l, ra, rr, leaf, order;
+	s64 imm;
+
+	if (!is_any(scev, id, &l, &rr))
+		return false;
+	reg = scev_reg_for_read(env, loop_entry, r);
+	if (!reg || !is_widenable_reg_type(reg))
+		return false;
+
+	scev->stack_sz = 0;
+	expr_stack_push(scev, id);
+	while (expr_next(scev, &id, &order)) {
+		if (order & DEPTH_LIMIT)
+			return false;
+		if (!(order & PRE) || is_any(scev, id, &l, &rr))
+			continue;
+		if (is_imm(scev, id, &imm)) {
+			if (reg->type != SCALAR_VALUE)
+				return false;
+		} else if (is_reg(scev, id, &leaf)) {
+			/* Other leaves must denote loop-invariant registers. */
+			if (leaf != r &&
+			    !(is_reg(scev, header_env->reg2scev[leaf], &ra) && ra == leaf))
+				return false;
+			leaf_reg = scev_reg_for_read(env, loop_entry, leaf);
+			if (!leaf_reg || !bpf_reg_union_compatible(reg, leaf_reg))
+				return false;
+		} else {
+			return false;
+		}
+	}
+	return true;
+}
+
+struct bounds {
+	struct cnum64 range;
+	u16 base;
+	u16 step;
+};
+
+static bool is_simple_linear(struct scev *scev, u32 id, u32 *base_reg, s64 *slope_imm)
+{
+	u32 base, slope;
+
+	return is_linear(scev, id, &base, &slope) &&
+	       is_reg(scev, base, base_reg) &&
+	       is_imm(scev, slope, slope_imm) &&
+	       *slope_imm <= S16_MAX &&
+	       *slope_imm >= S16_MIN &&
+	       *slope_imm != 0;
+}
+
+/*
+ * Compute the range an induction variable in `reg` spans over the loop.
+ * If the computation overflows s64 the range is unbounded, only the
+ * power-of-two alignment survives the wraparound.
+ */
+static void linear_bounds(struct bpf_reg_state *reg, u32 max_header_count, s64 slope,
+			  struct bounds *out)
+{
+	s64 slope_abs = slope < 0 ? -slope : slope;
+	s64 min_val = reg_smin(reg);
+	s64 max_val = reg_smax(reg);
+	s64 total_change;
+	bool overflow;
+
+	overflow = check_mul_overflow(slope, (s64)max_header_count - 1, &total_change) ||
+		   (slope > 0 && check_add_overflow(max_val, total_change, &max_val)) ||
+		   (slope < 0 && check_add_overflow(min_val, total_change, &min_val));
+	/*
+	 * If the entry value is a single point the value set is 'v + slope * k',
+	 * so the step is |slope|. Otherwise, only the power-of-two alignment
+	 * shared by the entry value and the slope.
+	 */
+	if (cnum64_is_const(reg->r64) && !overflow) {
+		out->step = slope_abs;
+		out->base = imod(min_val, out->step);
+	} else {
+		out->step = 1u << min_t(u32, tnum_alignment(reg->var_off), __ffs(slope_abs));
+		out->base = 0;
+	}
+	out->range = overflow ? CNUM64_UNBOUNDED : cnum64_from_srange(min_val, max_val);
+}
+
+static int mark_expr_precise(struct bpf_verifier_env *env, struct bpf_func_state *st, u32 id,
+			     unsigned long *precise_regs)
+{
+	struct scev *scev = env->scev;
+	struct bpf_reg_state *reg;
+	u32 r, order;
+
+	scev->stack_sz = 0;
+	expr_stack_push(scev, id);
+	while (expr_next(scev, &id, &order)) {
+		if (order & DEPTH_LIMIT) {
+			verifier_bug(env, "scev precision marking exceeds expression depth limit");
+			return -EFAULT;
+		}
+		if (!(order & PRE) || !is_reg(scev, id, &r))
+			continue;
+		/* Synthetic values have no spilled register to mark precise. */
+		reg = scev_reg_for_linear(st, r);
+		if (reg && reg->type == SCALAR_VALUE)
+			__set_bit(r, precise_regs);
+	}
+	return 0;
+}
+
+int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
+			   struct bpf_loop_iters *iters)
+{
+	struct bpf_func_state *cur_func = st->frame[st->curframe];
+	struct bpf_insn_aux_data *aux = env->insn_aux_data;
+	struct bpf_verifier_log *log = &env->log;
+	struct scev *scev = env->scev;
+	struct env *header_env;
+	struct linear_latch latch, best;
+	struct bpf_backedge *backedge;
+	struct bpf_loop_iters tmp;
+	struct bpf_loop *loop;
+	int insn_idx = st->insn_idx;
+	bool found = false;
+	int linear_latch;
+	u32 r, base_reg;
+	int latch_idx;
+	s64 slope_imm;
+	int err, j;
+
+	if (!env->allow_uninit_stack)
+		return 0;
+
+	/*
+	 * If insn_idx is a loop header for a reducible loop with a single backedge.
+	 * loop is NULL for secondary entries to irreducible loops.
+	 */
+	loop = aux[insn_idx].loop;
+	if (!loop || loop->irreducible || loop->backedges_cnt != 1 || loop->backedges_overflow ||
+	    loop->exits_overflow) {
+		if (log->level & BPF_LOG_LEVEL2)
+			bpf_log(log, "loop header at %d, unsupported loop:%s%s%s\n", insn_idx,
+				!loop || loop->irreducible ? " irreducible" : "",
+				loop && loop->backedges_cnt > 1 ? " multiple backedges" : "",
+				loop && loop->exits_overflow ? " too many exits" : "");
+		return 0;
+	}
+
+	/* If this backedge has a latch */
+	backedge = &loop->backedges[0];
+	if (!backedge->latches_cnt) {
+		if (log->level & BPF_LOG_LEVEL2)
+			bpf_log(log, "loop header at %d, latch not identified\n", insn_idx);
+		return 0;
+	}
+
+	/*
+	 * Every condition dominating the backedge bounds the number of
+	 * iterations from above, keep the smallest bound.
+	 */
+	for (j = 0; j < backedge->latches_cnt; j++) {
+		latch_idx = backedge->latches[j];
+		linear_latch = match_linear_latch(env, insn_idx, latch_idx, &latch);
+		if (linear_latch < 0)
+			return linear_latch;
+		if (!linear_latch) {
+			if (log->level & BPF_LOG_LEVEL2)
+				bpf_log(log, "loop header at %d, non-linear %s at %d\n",
+					insn_idx, j ? "exit" : "latch", latch_idx);
+			continue;
+		}
+		if (!compute_max_iters(env, cur_func, &latch, &tmp)) {
+			if (log->level & BPF_LOG_LEVEL2)
+				bpf_log(log,
+					"loop header at %d, can't compute iterations count at %d\n",
+					insn_idx, latch_idx);
+			continue;
+		}
+		if (!found || tmp.max_header_count < iters->max_header_count) {
+			iters->max_header_count = tmp.max_header_count;
+			best = latch;
+		}
+		iters->min_header_count = found ? min(iters->min_header_count, tmp.min_header_count)
+						: tmp.min_header_count;
+		found = true;
+	}
+	if (!found) {
+		if (log->level & BPF_LOG_LEVEL2)
+			bpf_log(log, "loop header at %d, can't compute iterations count\n", insn_idx);
+		return 0;
+	}
+	latch = best;
+
+	if (iters->max_header_count == 0) {
+		if (log->level & BPF_LOG_LEVEL2)
+			bpf_log(log, "loop header at %d, 0 iterations count\n", insn_idx);
+		return 0;
+	}
+
+	if (iters->max_header_count == U32_MAX) {
+		if (log->level & BPF_LOG_LEVEL2)
+			bpf_log(log, "loop header at %d, inf iterations count\n", insn_idx);
+		return 0;
+	}
+
+	if (log->level & BPF_LOG_LEVEL2) {
+		bpf_log(log, "loop header at %d, header_count is ", insn_idx);
+		if (iters->min_header_count == iters->max_header_count)
+			bpf_log(log, "%u ", iters->max_header_count);
+		else
+			bpf_log(log, "[%u..%u] ", iters->min_header_count, iters->max_header_count);
+		bpf_log(log, "\n");
+	}
+
+	err = bpf_live_stack_query_init(env, st);
+	if (err)
+		return err;
+
+	header_env = find_header_env(scev, insn_idx);
+	for (r = 0; r < REGS_NUM; r++) {
+		struct bpf_reg_state *reg;
+		u32 ra, r_scev, r_expr;
+
+		if (!scev_reg_alive(env, st, r))
+			continue;
+		/*
+		 * A heuristic:
+		 * Ignore SCEVs computed for unbound values and treat those as widened.
+		 * This is poor man's modelling of e.g. bpf_probe_read_kernel() effect
+		 * on stack, when called from a subprogram. Currently this would produce
+		 * an unknown ('?') SCEV as a summary of a subprogram call effects.
+		 * Ideally, this should be computed as an OPAQUE value.
+		 * However, that would require tracking STACK_MISC writes in the
+		 * liveness analysis. Not implemented yet.
+		 */
+		if (is_unbound_scev_reg(env, cur_func, r))
+			continue;
+
+		r_scev = header_env->reg2scev[r];
+		reg = scev_reg_for_linear(cur_func, r);
+		/* If SCEV for r is (linear <reg> <slope>) */
+		if (is_simple_linear(scev, r_scev, &base_reg, &slope_imm) &&
+		    reg && is_widenable_reg_type(reg)) {
+			continue;
+		}
+
+		/* rA = rA, loop does not change this reg */
+		if (is_reg(scev, r_scev, &ra) && r == ra)
+			continue;
+
+		/* (any 1 (any 2 (any 3 4))) */
+		if (is_widenable_any(env, cur_func, header_env, r, r_scev)) {
+			continue;
+		}
+
+		if (log->level & BPF_LOG_LEVEL2) {
+			r_expr = header_env->reg2expr[r];
+			bpf_log(log, "loop header at %d, can't widen ", insn_idx);
+			log_reg(env, r);
+			bpf_log(log, ", expr is ");
+			log_expr(env, r_expr);
+			bpf_log(log, "\n");
+		}
+		return 0;
+	}
+
+	bitmap_zero(iters->precise_regs, REGS_NUM);
+	err = mark_expr_precise(env, cur_func, latch.base_expr, iters->precise_regs);
+	if (err)
+		return err;
+	err = mark_expr_precise(env, cur_func, latch.step_expr, iters->precise_regs);
+	if (err)
+		return err;
+	err = mark_expr_precise(env, cur_func, latch.bound_expr, iters->precise_regs);
+	if (err)
+		return err;
+
+	return 1;
+}
+
+/* bpf_compute_loop_iters() checked that every leaf can be unioned into acc. */
+static int union_any_reg(struct bpf_verifier_env *env, struct bpf_func_state *loop_entry,
+			 struct bpf_reg_state *acc, u32 id)
+{
+	struct bpf_reg_state *tmp = &env->fake_reg[0];
+	const struct bpf_reg_state *leaf_reg;
+	struct scev *scev = env->scev;
+	u32 l, r, leaf, order;
+	s64 imm;
+	int err;
+
+	scev->stack_sz = 0;
+	expr_stack_push(scev, id);
+	while (expr_next(scev, &id, &order)) {
+		if (order & DEPTH_LIMIT) {
+			verifier_bug(env, "scev ANY union exceeds expression depth limit");
+			return -EFAULT;
+		}
+		if (!(order & PRE) || is_any(scev, id, &l, &r))
+			continue;
+		if (is_imm(scev, id, &imm)) {
+			bpf_mark_reg_known_scalar(tmp, imm);
+			leaf_reg = tmp;
+		} else if (is_reg(scev, id, &leaf)) {
+			leaf_reg = scev_reg_for_read(env, loop_entry, leaf);
+			if (verifier_bug_if(!leaf_reg, env, "unreadable SCEV ANY leaf %u", leaf))
+				return -EFAULT;
+		} else {
+			verifier_bug(env, "scev ANY union has an unsupported leaf");
+			return -EFAULT;
+		}
+		err = bpf_reg_union(env, acc, leaf_reg);
+		if (err)
+			return err;
+	}
+	return 0;
+}
+
+int bpf_widen_scev_regs(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
+			struct bpf_verifier_state *loop_entry, struct bpf_loop_iters *iters)
+{
+	struct bpf_func_state *cur_func = st->frame[st->curframe];
+	struct bpf_func_state *entry_func = loop_entry->frame[loop_entry->curframe];
+	struct bpf_verifier_log *log = &env->log;
+	struct scev *scev = env->scev;
+	struct bpf_reg_state *reg;
+	struct env *header_env;
+	struct bounds bounds;
+	u32 r, base_reg, r_expr, a, b;
+	int insn_idx = st->insn_idx;
+	s64 slope_imm;
+	int err;
+
+	for_each_set_bit(r, iters->precise_regs, REGS_NUM) {
+		if (r < MAX_BPF_REG)
+			bpf_bt_set_frame_reg(&env->bt, st->curframe, r);
+		else
+			bpf_bt_set_frame_slot(&env->bt, st->curframe, r - MAX_BPF_REG);
+	}
+	err = bpf_mark_chain_precision(env, st, -1, NULL);
+	if (err)
+		return err;
+
+	header_env = find_header_env(scev, insn_idx);
+	for (r = 0; r < REGS_NUM; r++) {
+		if (!scev_reg_alive(env, st, r) || is_unbound_scev_reg(env, cur_func, r))
+			continue;
+
+		r_expr = header_env->reg2scev[r];
+		/* If SCEV for r is (linear <reg> <slope>)*/
+		if (is_simple_linear(scev, r_expr, &base_reg, &slope_imm) &&
+		    base_reg == r) {
+			err = scev_reg_for_write(env, cur_func, r, &reg);
+			if (err)
+				return err;
+			linear_bounds(reg, iters->max_header_count, slope_imm, &bounds);
+			if (log->level & BPF_LOG_LEVEL2) {
+				bpf_log(log, "loop header at %d, widening ", insn_idx);
+				log_reg(env, r);
+				bpf_log(log, " to ");
+				bpf_verbose_snum(env, cnum64_smin(bounds.range));
+				bpf_log(log, "..");
+				bpf_verbose_snum(env, cnum64_smax(bounds.range));
+				bpf_log(log, " step %u\n", bounds.step);
+			}
+			scratch_widened_reg_id(env, reg);
+			err = bpf_set_reg_range(env, reg, bounds.range, bounds.base, bounds.step);
+			if (err)
+				return err;
+			mark_scev_reg_scratched(env, r);
+		} else if (is_any(scev, r_expr, &a, &b)) {
+			err = scev_reg_for_write(env, cur_func, r, &reg);
+			if (err)
+				return err;
+			err = union_any_reg(env, entry_func, reg, r_expr);
+			if (err)
+				return err;
+			if (log->level & BPF_LOG_LEVEL2) {
+				bpf_log(log, "loop header at %d, widening ", insn_idx);
+				log_reg(env, r);
+				bpf_log(log, " to ");
+				bpf_verbose_snum(env, reg_smin(reg));
+				bpf_log(log, "..");
+				bpf_verbose_snum(env, reg_smax(reg));
+				bpf_log(log, " step %u\n", reg->step);
+			}
+			scratch_widened_reg_id(env, reg);
+			mark_scev_reg_scratched(env, r);
+		}
+	}
+	return 1;
+}
+
+int bpf_clamp_scev_regs(struct bpf_verifier_env *env, struct bpf_func_state *cur_func_state, u32 insn_idx,
+			struct bpf_verifier_state *entry_state, u32 max_header_count)
+{
+	struct bpf_func_state *entry_st = entry_state->frame[entry_state->curframe];
+	struct bpf_verifier_log *log = &env->log;
+	struct bpf_reg_state *reg, *entry_reg;
+	struct scev *scev = env->scev;
+	struct env *header_env;
+	struct bounds bounds;
+	u32 r, base_reg;
+	s64 slope_imm;
+	int err;
+
+	if (entry_state->curframe != cur_func_state->frameno) {
+		verifier_bug(env, "clamping registers for a wrong frame: %d vs %d\n",
+			     entry_state->curframe, cur_func_state->frameno);
+		return -EFAULT;
+	}
+
+	header_env = find_header_env(scev, insn_idx);
+	for (r = 0; r < REGS_NUM; r++) {
+		/* If SCEV for r is (linear <reg> <slope>)*/
+		if (!is_simple_linear(scev, header_env->reg2scev[r], &base_reg, &slope_imm) ||
+		    base_reg != r)
+			continue;
+
+		entry_reg = scev_reg_for_linear(entry_st, r);
+		reg = scev_reg_for_linear(cur_func_state, r);
+		if (!entry_reg || !reg || entry_reg->type == NOT_INIT || reg->type == NOT_INIT ||
+		    is_unbound_scev_reg(env, entry_st, r))
+			continue;
+
+		linear_bounds(entry_reg, max_header_count, slope_imm, &bounds);
+		bounds.range = cnum64_intersect(reg->r64, bounds.range);
+		if (cnum64_is_empty(bounds.range)) {
+			verifier_bug(env, "scev clamp produced empty range for r%d", r);
+			return -EFAULT;
+		}
+		if (log->level & BPF_LOG_LEVEL2) {
+			bpf_log(log, "loop header at %d, clamping ", insn_idx);
+			log_reg(env, r);
+			bpf_log(log, " to ");
+			bpf_verbose_snum(env, cnum64_smin(bounds.range));
+			bpf_log(log, "..");
+			bpf_verbose_snum(env, cnum64_smax(bounds.range));
+			bpf_log(log, " step %u\n", bounds.step);
+		}
+		scratch_widened_reg_id(env, reg);
+		err = bpf_set_reg_range(env, reg, bounds.range, bounds.base, bounds.step);
+		if (err)
+			return err;
+		mark_scev_reg_scratched(env, r);
+	}
+	return 0;
+}
diff --git a/kernel/bpf/states.c b/kernel/bpf/states.c
index 470bff8d8dc8..c62ea992967b 100644
--- a/kernel/bpf/states.c
+++ b/kernel/bpf/states.c
@@ -936,6 +936,27 @@ static bool refsafe(struct bpf_verifier_state *old, struct bpf_verifier_state *c
 	return true;
 }
 
+static bool loop_stack_safe(struct bpf_verifier_env *env, struct bpf_func_state *old,
+			    struct bpf_func_state *cur)
+{
+	struct loop_stack_entry *old_stack = old->loop_stack;
+	struct loop_stack_entry *cur_stack = cur->loop_stack;
+	u32 i;
+
+	if (old->loop_stack_cnt != cur->loop_stack_cnt)
+		return false;
+
+	for (i = 0; i < old->loop_stack_cnt; i++) {
+		if (old_stack[i].loop_id == cur_stack[i].loop_id &&
+		    old_stack[i].max_header_count == cur_stack[i].max_header_count &&
+		    old_stack[i].terminates == cur_stack[i].terminates)
+			continue;
+		return false;
+	}
+
+	return true;
+}
+
 /* compare two verifier states
  *
  * all states stored in state_list are known to be valid, since
@@ -986,6 +1007,9 @@ static bool func_states_equal(struct bpf_verifier_env *env, struct bpf_func_stat
 	if (!stack_arg_safe(env, old, cur, &env->idmap_scratch, exact))
 		return false;
 
+	if (!loop_stack_safe(env, old, cur))
+		return false;
+
 	return true;
 }
 
@@ -1032,6 +1056,7 @@ static bool states_equal(struct bpf_verifier_env *env,
 		if (!func_states_equal(env, old->frame[i], cur->frame[i], insn_idx, exact))
 			return false;
 	}
+
 	return true;
 }
 
@@ -1314,15 +1339,40 @@ int bpf_split_cur_state(struct bpf_verifier_env *env)
 	return 0;
 }
 
+/* Force a checkpoint upon reaching a loop header for a terminating loop. */
+static bool need_loop_checkpoint(struct bpf_verifier_env *env, int insn_idx)
+{
+	struct bpf_verifier_state *cur = env->cur_state;
+	struct bpf_func_state *frame = cur->frame[cur->curframe];
+	struct loop_stack_entry *top;
+
+	if (bpf_loop_at_index(env, insn_idx) != insn_idx || !frame->loop_stack_cnt)
+		return false;
+	top = &frame->loop_stack[frame->loop_stack_cnt - 1];
+	return top->loop_id == insn_idx && top->terminates;
+}
+
+static struct loop_stack_entry *current_loop(struct bpf_verifier_state *st)
+{
+	struct bpf_func_state *frame = st->frame[st->curframe];
+
+	if (frame->loop_stack_cnt == 0)
+		return NULL;
+	return &frame->loop_stack[frame->loop_stack_cnt - 1];
+}
+
 int bpf_is_state_visited(struct bpf_verifier_env *env, int insn_idx)
 {
+	struct loop_stack_entry *old_loop, *cur_loop;
 	struct bpf_verifier_state_list *sl;
 	struct bpf_verifier_state *cur = env->cur_state;
-	bool force_new_state, add_new_state, loop;
+	bool force_new_state, add_new_state, loop, loop_checkpoint;
 	int n, err, states_cnt = 0;
 	struct list_head *pos, *tmp, *head;
 
+	loop_checkpoint = need_loop_checkpoint(env, insn_idx);
 	force_new_state = env->test_state_freq || bpf_is_force_checkpoint(env, insn_idx) ||
+			  loop_checkpoint ||
 			  /* Avoid accumulating infinitely long jmp history */
 			  cur->jmp_history_cnt > 40;
 
@@ -1353,10 +1403,10 @@ int bpf_is_state_visited(struct bpf_verifier_env *env, int insn_idx)
 			continue;
 
 		if (sl->state.branches) {
-			struct bpf_func_state *frame = sl->state.frame[0];
+			struct bpf_func_state *old_top_frame = sl->state.frame[0];
 
-			if (frame->in_async_callback_fn &&
-			    frame->async_entry_cnt != cur->frame[0]->async_entry_cnt) {
+			if (old_top_frame->in_async_callback_fn &&
+			    old_top_frame->async_entry_cnt != cur->frame[0]->async_entry_cnt) {
 				/* Different async_entry_cnt means that the verifier is
 				 * processing another entry into async callback.
 				 * Seeing the same state is not an indication of infinite
@@ -1455,6 +1505,33 @@ int bpf_is_state_visited(struct bpf_verifier_env *env, int insn_idx)
 				}
 				goto skip_inf_loop_check;
 			}
+			/*
+			 * If old state belongs to a control flow loop that we know terminates,
+			 * it should be safe to prune current state. However, account for the
+			 * following situation:
+			 *
+			 *   for (;;) {
+			 *     for (i = 0; i < 10; i++)
+			 *       ...
+			 *   }
+			 *
+			 * Here a backedge to a non-terminating outer loop is dominated by an exit
+			 * from the inner loop. Pruning at a re-entry to the inner loop therefore
+			 * would hide the fact that outer loop is non-terminating.
+			 * Hence, only allow pruning states within the same entry state.
+			 */
+			old_loop = current_loop(&sl->state);
+			cur_loop = current_loop(cur);
+			if (old_loop && cur_loop &&
+			    old_loop->terminates && cur_loop->terminates &&
+			    old_loop->entry_state == cur_loop->entry_state) {
+				if (states_equal(env, &sl->state, cur, RANGE_WITHIN)) {
+					loop = true;
+					goto hit;
+				}
+				goto skip_inf_loop_check;
+			}
+
 			/* attempt to detect infinite loop to avoid unnecessary doomed work */
 			if (states_maybe_looping(&sl->state, cur) &&
 			    states_equal(env, &sl->state, cur, EXACT) &&
@@ -1613,7 +1690,8 @@ int bpf_is_state_visited(struct bpf_verifier_env *env, int insn_idx)
 		 * Use bigger 'n' for checkpoints because evicting checkpoint states
 		 * too early would hinder iterator convergence.
 		 */
-		n = bpf_is_force_checkpoint(env, insn_idx) && sl->state.branches > 0 ? 64 : 3;
+		n = (bpf_is_force_checkpoint(env, insn_idx) || loop_checkpoint) &&
+		    sl->state.branches > 0 ? 64 : 3;
 		if (sl->miss_cnt > sl->hit_cnt * n + n) {
 			/* the state is unlikely to be useful. Remove it to
 			 * speed up verification
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 60287049cfe4..a597a8c18972 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -1681,6 +1681,7 @@ static void free_func_state(struct bpf_func_state *state)
 {
 	if (!state)
 		return;
+	kfree(state->loop_stack);
 	kfree(state->stack_arg_regs);
 	kfree(state->stack);
 	kfree(state);
@@ -1717,6 +1718,10 @@ static int copy_func_state(struct bpf_func_state *dst,
 	memcpy(dst, src, offsetof(struct bpf_func_state, stack));
 	/* Instruction accounting is path-local, not part of verifier state. */
 	dst->insns_subtotal = 0;
+	dst->loop_stack = copy_array(dst->loop_stack, src->loop_stack, src->loop_stack_cnt,
+				     sizeof(*src->loop_stack), GFP_KERNEL_ACCOUNT);
+	if (!dst->loop_stack)
+		return -ENOMEM;
 	return copy_stack_state(dst, src);
 }
 
@@ -19916,6 +19921,178 @@ static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state)
 	return -EFAULT;
 }
 
+/*
+ * Push the loop entered at insn_idx onto the stack. Usually this adds a single
+ * entry on top of its enclosing loop. However, an edge may enter an inner loop
+ * directly, bypassing the header(s) of its enclosing loop(s), e.g.:
+ *
+ *   1: for (...):       // enclosing loop, header at 1
+ *   2:   for (...):     // inner loop, header at 2
+ *        ...
+ *   3: if ...:
+ *        goto 2b;       // enters loop 2 without going through header 1
+ *
+ * Such a bypass makes the enclosing loop irreducible, so its header is missing
+ * from the stack and both headers (1) and (2) need to be pushed onto stack.
+ * Only the innermost loop (the one actually entered at insn_idx) carries SCEV bounds;
+ * the bypassed ancestors are irreducible and pushed as non-terminating.
+ *
+ * Assumes loop_stack_pop() has already truncated the stack to the common ancestor
+ * of the bpf_loop_at_index(env->insn_idx) and whatever was at the top of the loop stack.
+ */
+static int loop_stack_push(struct bpf_verifier_env *env, bool *pushed)
+{
+	struct bpf_insn_aux_data *aux = env->insn_aux_data;
+	struct bpf_func_state *frame = cur_func(env);
+	struct loop_stack_entry *loop_stack = frame->loop_stack;
+	u32 missing_headers[LOOP_STACK_SIZE];
+	u32 cnt = frame->loop_stack_cnt;
+	u32 num_missing = 0;
+	int h;
+
+	*pushed = false;
+
+	for (h = bpf_loop_at_index(env, env->insn_idx); h >= 0; h = aux[h].loop_header, num_missing++) {
+		if (cnt && loop_stack[cnt - 1].loop_id == h)
+			break;
+		if (num_missing == LOOP_STACK_SIZE)
+			goto e2big;
+		missing_headers[num_missing] = h;
+	}
+
+	if (num_missing == 0)
+		return 0;
+	if (cnt + num_missing > LOOP_STACK_SIZE)
+		goto e2big;
+	frame->loop_stack = realloc_array(frame->loop_stack, cnt, cnt + num_missing,
+					 sizeof(*frame->loop_stack));
+	if (!frame->loop_stack)
+		return -ENOMEM;
+	loop_stack = frame->loop_stack;
+
+	for (; num_missing; num_missing--, cnt++) {
+		h = missing_headers[num_missing - 1];
+		loop_stack[cnt] = (struct loop_stack_entry){ .loop_id = h };
+		if (env->log.level & BPF_LOG_LEVEL2)
+			verbose(env, "entering loop %d\n", h);
+		*pushed = true;
+	}
+	frame->loop_stack_cnt = cnt;
+	return 0;
+
+e2big:
+	verbose(env, "Too many nested loops (%d/%d) at %d\n", cnt, num_missing, env->insn_idx);
+	return -E2BIG;
+}
+
+/*
+ * An exit from a loop can cross several nested loops, e.g.:
+ *
+ *   1: for (...):
+ *   2:   for (...):
+ *   3:     for (...):
+ *            if ...:    // before goto the loop stack is [1, 2, 3 <top>]
+ *              goto 1b; // after goto it should become [1]
+ */
+static int loop_stack_pop(struct bpf_verifier_env *env)
+{
+	struct bpf_insn_aux_data *aux = env->insn_aux_data;
+	struct bpf_func_state *frame = cur_func(env);
+	int new_cnt = 0;
+	int h, i;
+
+	/*
+	 * Walk the loop nest of insn_idx outwards (innermost first) and stop at
+	 * the first loop that is present on the stack - that loop is the common
+	 * ancestor and becomes the new top. If no enclosing loop is on the
+	 * stack, drop all loops.
+	 */
+	for (h = bpf_loop_at_index(env, env->insn_idx); h >= 0; h = aux[h].loop_header) {
+		for (i = frame->loop_stack_cnt - 1; i >= 0; i--) {
+			if (frame->loop_stack[i].loop_id == h) {
+				new_cnt = i + 1;
+				goto pop;
+			}
+		}
+	}
+
+pop:
+	if (env->log.level & BPF_LOG_LEVEL2)
+		for (i = frame->loop_stack_cnt - 1; i >= new_cnt; i--)
+			verbose(env, "exiting loop %d\n", frame->loop_stack[i].loop_id);
+	frame->loop_stack_cnt = new_cnt;
+	return 0;
+}
+
+static int maybe_clamp_scev_regs(struct bpf_verifier_env *env)
+{
+	struct bpf_func_state *frame = cur_func(env);
+	struct loop_stack_entry *entry;
+
+	if (frame->loop_stack_cnt == 0) {
+		verifier_bug(env, "%s: loop stack empty at %d", __FUNCTION__, env->insn_idx);
+		return -EFAULT;
+	}
+
+	entry = &frame->loop_stack[frame->loop_stack_cnt - 1];
+	if (!entry->terminates)
+		return 0;
+
+	return bpf_clamp_scev_regs(env, frame, env->insn_idx, entry->entry_state,
+				   entry->max_header_count);
+}
+
+static int handle_loop_entry_exit(struct bpf_verifier_env *env)
+{
+	struct bpf_verifier_state *cur = env->cur_state;
+	struct bpf_func_state *frame = cur_func(env);
+	struct loop_stack_entry *entry;
+	struct bpf_loop_iters iters;
+	bool pushed, terminates;
+	int err;
+
+	err = loop_stack_pop(env);
+	if (err)
+		return err;
+
+	err = loop_stack_push(env, &pushed);
+	if (err)
+		return err;
+
+	if (frame->loop_stack_cnt == 0)
+		return 0;
+
+	entry = &frame->loop_stack[frame->loop_stack_cnt - 1];
+	if (env->insn_idx != entry->loop_id)
+		return 0;
+
+	/* If nothing was pushed and env->insn_idx is a loop header, we've taken a backedge. */
+	if (!pushed)
+		return maybe_clamp_scev_regs(env);
+
+	err = bpf_compute_loop_iters(env, env->cur_state, &iters);
+	if (err < 0)
+		return err;
+
+	terminates = err == 1;
+	if (!terminates)
+		return 0;
+
+	/* Create a loop_entry checkpoint before widening */
+	err = bpf_split_cur_state(env);
+	if (err)
+		return err;
+
+	entry->min_header_count = iters.min_header_count;
+	entry->max_header_count = iters.max_header_count;
+	entry->entry_state = cur->parent;
+	entry->terminates = true;
+	err = bpf_widen_scev_regs(env, cur, cur->parent, &iters);
+	if (err < 0)
+		return err;
+	return 0;
+}
+
 static int do_check(struct bpf_verifier_env *env)
 {
 	bool pop_log = !(env->log.level & BPF_LOG_LEVEL2);
@@ -19930,6 +20107,12 @@ static int do_check(struct bpf_verifier_env *env)
 		struct bpf_insn_aux_data *insn_aux;
 		int err;
 
+		if (signal_pending(current))
+			return -EAGAIN;
+
+		if (need_resched())
+			cond_resched();
+
 		/* reset current history entry on each new instruction */
 		env->cur_hist_ent = NULL;
 
@@ -19976,6 +20159,15 @@ static int do_check(struct bpf_verifier_env *env)
 			}
 		}
 
+		/*
+		 * Possibly widen the registers before creating a checkpoint
+		 * in bpf_is_state_visited(). The next loop iteration will
+		 * have a chance to hit this checkpoint and converge.
+		 */
+		err = handle_loop_entry_exit(env);
+		if (err)
+			return err;
+
 		if (bpf_is_prune_point(env, env->insn_idx)) {
 			err = bpf_is_state_visited(env, env->insn_idx);
 			if (err < 0)
@@ -20001,12 +20193,6 @@ static int do_check(struct bpf_verifier_env *env)
 				return err;
 		}
 
-		if (signal_pending(current))
-			return -EAGAIN;
-
-		if (need_resched())
-			cond_resched();
-
 		if (env->log.level & BPF_LOG_LEVEL2 && do_print_state) {
 			verbose(env, "\nfrom %d to %d%s:",
 				env->prev_insn_idx, env->insn_idx,
@@ -20016,6 +20202,13 @@ static int do_check(struct bpf_verifier_env *env)
 			do_print_state = false;
 		}
 
+		if (bpf_loop_at_index(env, env->insn_idx) >= 0 &&
+		    cur_func(env)->loop_stack_cnt == 0) {
+			verifier_bug(env, "loop stack empty at %d, while inside the loop %d\n",
+				     env->insn_idx, bpf_loop_at_index(env, env->insn_idx));
+			return -EFAULT;
+		}
+
 		if (env->log.level & BPF_LOG_LEVEL) {
 			if (verifier_state_scratched(env))
 				print_insn_state(env, state, state->curframe);
diff --git a/tools/testing/selftests/bpf/progs/verifier_gotox.c b/tools/testing/selftests/bpf/progs/verifier_gotox.c
index dc7baa0f2458..84302a122ccf 100644
--- a/tools/testing/selftests/bpf/progs/verifier_gotox.c
+++ b/tools/testing/selftests/bpf/progs/verifier_gotox.c
@@ -511,7 +511,7 @@ __naked void too_many_gotox_edges(void)
 
 SEC("socket")
 __description("gotox-edges-at-limit")
-__success __retval(0)
+__failure __msg("Too many nested loops")
 __naked void gotox_edges_at_limit(void)
 {
 	asm volatile (
@@ -519,6 +519,10 @@ __naked void gotox_edges_at_limit(void)
 		 * 1000 gotox * 1000 targets = 1,000,000 CFG edges. At run
 		 * time, each block loads the next table entry and jumps to
 		 * the following block, so the program terminates.
+		 *
+		 * Every block can jump to every later block, so the CFG is
+		 * a nest of ~1000 irreducible loops, deeper than the
+		 * verifier's loop stack; the program is rejected.
 		 */
 		GOTOX_TABLE_BEGIN(1000)
 		".rept 1000;"

-- 
2.53.0

Re: [PATCH bpf-next v3 34/51] bpf: use SCEV to widen bounded loops

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 4159 bytes --]

> bpf: use SCEV to widen bounded loops

TL;DR:
- Loop widening may turn stack accesses variable-offset, so programs that
  loaded before (dynptr loops, pointer spills) could be rejected until a later
  patch fixes it
- nit: __FUNCTION__ vs __func__

> diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
> index 2d2a84156180..05dd8e439609 100644
> --- a/kernel/bpf/scev.c
> +++ b/kernel/bpf/scev.c
> @@ -1638,3 +1646,1034 @@ int bpf_init_scev(struct bpf_verifier_env *env)
[ ... ]

> +int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
> +			   struct bpf_loop_iters *iters)
[ ... ]

> +		r_scev = header_env->reg2scev[r];
> +		reg = scev_reg_for_linear(cur_func, r);
> +		/* If SCEV for r is (linear <reg> <slope>) */
> +		if (is_simple_linear(scev, r_scev, &base_reg, &slope_imm) &&
> +		    reg && is_widenable_reg_type(reg)) {
> +			continue;
> +		}
> +
> +		/* rA = rA, loop does not change this reg */
> +		if (is_reg(scev, r_scev, &ra) && r == ra)
> +			continue;
> +
> +		/* (any 1 (any 2 (any 3 4))) */
> +		if (is_widenable_any(env, cur_func, header_env, r, r_scev)) {
> +			continue;
> +		}
[ ... ]

Can widening here break stack accesses that used to verify?

bpf_compute_loop_iters() accepts a linear or (any ...) register based only on
its SCEV shape and register type.  It does not look at how the loop uses the
register.  If the register is used to build a stack address (r2 = r10;
r2 += -16; r2 += r7), widening turns constant-offset stack accesses into
variable-offset ones, and that is handled less precisely in the main pass:

  - A variable-offset fill of a spilled pointer ends up in
    mark_reg_stack_read(), which falls back to mark_reg_unknown(), so the
    pointer becomes a scalar.

  - A variable-offset spill marks the slots STACK_MISC, so the stored value
    is lost.

  - dynptr, iterator, irq and res_spin_lock stack objects go through
    stack_slot_obj_get_spi(), which rejects a non-constant var_off with
    "has to be at a constant offset".

For example:

        *(u64 *)(r10 - 16) = r1;
        *(u64 *)(r10 - 8) = r1;
        r7 = 0;
        r8 = 0;
    1:  r2 = r10;
        r2 += -16;
        r2 += r7;
        r3 = *(u64 *)(r2 + 0);
        r0 = *(u32 *)(r3 + 0);
        if r8 != 0 goto 2f;
        r7 = 8;
    2:  r8 += 1;
        if r8 < 2 goto 1b;

Here r8 is (linear r8 1) and r7 is (any r7 8), so r7 is widened to {0,8}.
The fill through r2 now has a variable offset and yields a scalar, and the
load through r3 is rejected.  Before this patch each iteration was explored
with a constant r7 and the program verified.

The same happens with a C loop such as:

    for (i = 0; i < 4; i++)
        bpf_dynptr_from_xdp(ctx, 0, &dptrs[i]);

which now fails with "dynptr has to be at a constant offset".  Spilling
pointers through a widened stack pointer fails in the same way.

A later patch in the series, 42d577fd54e2 ("bpf: avoid widening registers
that hinder exact stack-slot tracking"), adds store_base_regs and makes
bpf_compute_loop_iters() refuse to widen such registers.  Until then, programs
that loaded before are rejected.  Should that check be part of this patch?

> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index 60287049cfe4..a597a8c18972 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
> @@ -19916,6 +19921,178 @@ static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state)
[ ... ]

> +static int maybe_clamp_scev_regs(struct bpf_verifier_env *env)
> +{
> +	struct bpf_func_state *frame = cur_func(env);
> +	struct loop_stack_entry *entry;
> +
> +	if (frame->loop_stack_cnt == 0) {
> +		verifier_bug(env, "%s: loop stack empty at %d", __FUNCTION__, env->insn_idx);
> +		return -EFAULT;
> +	}

this isn't a bug, but should __FUNCTION__ be __func__ here?  checkpatch
flags the gcc specific spelling, and the other verifier_bug() callers in
verifier.c use __func__.


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/37923187511

Re: [PATCH bpf-next v3 34/51] bpf: use SCEV to widen bounded loops

[kernel test robot]

Hi Eduard,

kernel test robot noticed the following build errors:

[auto build test ERROR on e1d84a37cba984388988d2f1ddc84561413f0db2]

url:    https://github.com/intel-lab-lkp/linux/commits/Eduard-Zingerman/bpf-represent-stack-access-effects-with-arg_access_info/20261009-034719
base:   e1d84a37cba984388988d2f1ddc84561413f0db2
patch link:    https://lore.kernel.org/r/20261009-scev-minimal-rebase-v3-34-bcba9bc53ac9%40gmail.com
patch subject: [PATCH bpf-next v3 34/51] bpf: use SCEV to widen bounded loops
config: i386-buildonly-randconfig-1002-20261010 (https://download.01.org/0day-ci/archive/20261010/202610100928.qjEH6jQv-lkp@intel.com/config)
compiler: gcc-14 (Debian 14.2.0-19) 14.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20261010/202610100928.qjEH6jQv-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202610100928.qjEH6jQv-lkp@intel.com/

All errors (new ones prefixed by >>):

   ld: kernel/bpf/scev.o: in function `compute_max_iters':
>> kernel/bpf/scev.c:2062:(.text+0x1193): undefined reference to `__udivdi3'
>> ld: kernel/bpf/scev.c:2067:(.text+0x11b7): undefined reference to `__udivdi3'
   ld: kernel/bpf/scev.c:2074:(.text+0x11e3): undefined reference to `__udivdi3'
   ld: kernel/bpf/scev.c:2075:(.text+0x1207): undefined reference to `__udivdi3'
   ld: kernel/bpf/scev.c:2081:(.text+0x123e): undefined reference to `__udivdi3'
   ld: kernel/bpf/scev.o:kernel/bpf/scev.c:2089: more undefined references to `__udivdi3' follow
   ld: kernel/bpf/scev.o: in function `compute_max_iters':
   kernel/bpf/scev.c:2141:(.text+0x1464): undefined reference to `__udivmoddi4'


vim +2062 kernel/bpf/scev.c

  1980	
  1981	/*
  1982	 * Loop with post-condition:
  1983	 *
  1984	 *    r0 = 0
  1985	 * l: ...                r0 ∈ [0,1,2] header executed 3 times
  1986	 *    r0 += 1            r0 ∈ [0,1,2]
  1987	 *    ...                r0 ∈ [1,2,3]
  1988	 *    if r0 != 3 goto l  r0 ∈ [1,2,3] backedge taken 2 times
  1989	 *    ...                r0 ∈ [3]
  1990	 *
  1991	 * SCEV at header: r0 = k
  1992	 * SCEV at latch:  r0 = 1 + k
  1993	 *
  1994	 * Loop with pre-condition:
  1995	 *
  1996	 *    r0 = 0
  1997	 * l: ...                r0 ∈ [0,1,2,3] header executed 4 times
  1998	 *    if r0 == 3 goto e  r0 ∈ [0,1,2,3] backedge taken 3 times
  1999	 *    r0 += 1            r0 ∈ [0,1,2]
  2000	 *    ...                r0 ∈ [1,2,3]
  2001	 *    goto l             r0 ∈ [1,2,3]
  2002	 * e: ...                r0 ∈ [3]
  2003	 *
  2004	 * SCEV at header: r0 = k
  2005	 * SCEV at latch:  r0 = k
  2006	 */
  2007	static bool compute_max_iters(struct bpf_verifier_env *env,
  2008				      struct bpf_func_state *st,
  2009				      struct linear_latch *latch,
  2010				      struct bpf_loop_iters *iters)
  2011	{
  2012		struct bpf_insn_aux_data *aux = env->insn_aux_data;
  2013		struct bpf_loop *loop = aux[bpf_loop_at_index(env, latch->insn_idx)].loop;
  2014		struct scev *scev = env->scev;
  2015		struct scev_value initial_val, step_val, bound_val;
  2016		u64 step, diff, bound, initial, backedge_taken_count;
  2017		u8 op = latch->op;
  2018		bool up, down;
  2019	
  2020		if (!eval_expr(env, scev, st, latch->base_expr, &initial_val) ||
  2021		    !eval_expr(env, scev, st, latch->step_expr, &step_val) ||
  2022		    !eval_expr(env, scev, st, latch->bound_expr, &bound_val))
  2023			return false;
  2024	
  2025		if (step_val.ptr_reg >= 0) {
  2026			if (env->log.level & BPF_LOG_LEVEL2) {
  2027				bpf_log(&env->log, "scev: latch step is a pointer ");
  2028				log_scev_value(env, st, &step_val);
  2029				bpf_log(&env->log, "\n");
  2030			}
  2031			return false;
  2032		}
  2033		if (!scev_value_compatible(env, st, &initial_val, &bound_val))
  2034			return false;
  2035	
  2036		initial = initial_val.value;
  2037		step = step_val.value;
  2038		bound = bound_val.value;
  2039	
  2040		if (step == 0)
  2041			return false;
  2042		down = (s64)step < 0;
  2043		up = (s64)step > 0;
  2044		diff = bound - initial;
  2045		switch (op) {
  2046		case BPF_JLT:
  2047			/*
  2048			 * E.g.: 3 + 2*i < 7 ⇔ 2*i < 4 ⇔ i ∈ [0,1].
  2049			 * Here the inequality describes a set of values `i` might take
  2050			 * while the latch condition remains true (the backedge is taken).
  2051			 * The backedge_taken_count is the size of this set.
  2052			 */
  2053			if (/*
  2054			     * If the condition is initially true, decreasing values won't
  2055			     * change that without underflow. If the condition is initially false,
  2056			     * the main pass would do a single iteration anyway.
  2057			     */
  2058			    down ||
  2059			    /* Is the latch condition false on the first iteration? */
  2060			    initial >= bound ||
  2061			    /* Count iterations satisfying the strict inequality, including i=0. */
> 2062			    check_add_overflow((diff - 1) / step, 1, &backedge_taken_count) ||
  2063			    /*
  2064			     * Check if the last iteration overflows the counter:
  2065			     *   initial + backedge_taken_count * step > U64_MAX
  2066			     */
> 2067			    backedge_taken_count > (U64_MAX - initial) / step)
  2068				return false;
  2069			break;
  2070		case BPF_JLE:
  2071			/* E.g.: 3 + 2*i ≤ 7 ⇔ 2*i ≤ 4 ⇔ i ∈ [0,1,2]. */
  2072			if (down || initial > bound ||
  2073			    /* Count iterations satisfying the non-strict inequality. */
  2074			    check_add_overflow(diff / step, 1, &backedge_taken_count) ||
  2075			    backedge_taken_count > (U64_MAX - initial) / step)
  2076				return false;
  2077			break;
  2078		case BPF_JGT:
  2079			/* E.g.: 7 - 2*i > 3 ⇔ 2*i < 4 ⇔ i ∈ [0,1]. */
  2080			if (up || initial <= bound ||
  2081			    check_add_overflow((-diff - 1) / -step, 1, &backedge_taken_count) ||
  2082			    /*
  2083			     * The last iteration underflows the counter if:
  2084			     *   initial + backedge_taken_count * step < 0 ⇔
  2085			     *             backedge_taken_count * step < -initial ⇔
  2086			     *                    backedge_taken_count > initial / -step
  2087			     * (direction flips because step < 0)
  2088			     */
  2089			    backedge_taken_count > initial / -step)
  2090				return false;
  2091			break;
  2092		case BPF_JGE:
  2093			/* E.g.: 7 - 2*i ≥ 3 ⇔ 2*i ≤ 4 ⇔ i ∈ [0,1,2]. */
  2094			if (up || initial < bound ||
  2095			    check_add_overflow(-diff / -step, 1, &backedge_taken_count) ||
  2096			    backedge_taken_count > initial / -step)
  2097				return false;
  2098			break;
  2099		case BPF_JSLT:
  2100			if (down || (s64)initial >= (s64)bound ||
  2101			    check_add_overflow((diff - 1) / step, 1, &backedge_taken_count) ||
  2102			    /*
  2103			     * Check if the last iteration overflows the signed counter:
  2104			     *   (s64)initial + backedge_taken_count * step > S64_MAX
  2105			     */
  2106			    backedge_taken_count > ((u64)S64_MAX - initial) / step)
  2107				return false;
  2108			break;
  2109		case BPF_JSLE:
  2110			if (down || (s64)initial > (s64)bound ||
  2111			    check_add_overflow(diff / step, 1, &backedge_taken_count) ||
  2112			    backedge_taken_count > ((u64)S64_MAX - initial) / step)
  2113				return false;
  2114			break;
  2115		case BPF_JSGT:
  2116			if (up || (s64)initial <= (s64)bound ||
  2117			    check_add_overflow((-diff - 1) / -step, 1, &backedge_taken_count) ||
  2118			    /*
  2119			     * The last iteration underflows the signed counter if:
  2120			     *   (s64)initial + backedge_taken_count * step < S64_MIN ⇔
  2121			     *                  backedge_taken_count * step < S64_MIN - (s64)initial ⇔
  2122			     *                         backedge_taken_count > initial - (u64)S64_MIN / -step
  2123			     *	 (direction flips because step < 0)
  2124			     */
  2125			    backedge_taken_count > (initial - (u64)S64_MIN) / -step)
  2126				return false;
  2127			break;
  2128		case BPF_JSGE:
  2129			if (up || (s64)initial < (s64)bound ||
  2130			    check_add_overflow(-diff / -step, 1, &backedge_taken_count) ||
  2131			    backedge_taken_count > (initial - (u64)S64_MIN) / -step)
  2132				return false;
  2133			break;
  2134		case BPF_JNE:
  2135			/*
  2136			 * For a decreasing counter flip the diff and step:
  2137			 *   7 - 2*i ≠ 3 ⇔ 2*i ≠ 7 - 3.
  2138			 */
  2139			diff = down ? -diff : diff;
  2140			step = down ? -step : step;
  2141			if (diff == 0 || diff % step)
  2142				return false;
  2143			backedge_taken_count = diff / step;
  2144			break;
  2145		default:
  2146			return false;
  2147		}
  2148	
  2149		/*
  2150		 * max_header_count is u32 and U32_MAX means "infinite",
  2151		 * check to avoid overflow below.
  2152		 */
  2153		if (backedge_taken_count >= U32_MAX - 1)
  2154			return false;
  2155	
  2156		/* The header executes once on entry and once for each taken backedge, hence +1. */
  2157		iters->max_header_count = backedge_taken_count + 1;
  2158		/*
  2159		 * The latch is a conditional jump with one jump target exiting the loop.
  2160		 * Linear latch is matched only if the loop has a single backedge.
  2161		 * The loop still, however can have multiple exits.
  2162		 * In such case, conservatively assume that non-latch exit can happen
  2163		 * at any iteration, thus setting minimal number of iterations as 0.
  2164		 */
  2165		iters->min_header_count = loop->exits_cnt == 1 ? iters->max_header_count : 0;
  2166		return true;
  2167	}
  2168	

--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [PATCH bpf-next v3 34/51] bpf: use SCEV to widen bounded loops

[Alexei Starovoitov]

On Sat, Oct 10, 2026 at 09:25 AM kernel test robot <lkp@intel.com> wrote:
>    ld: kernel/bpf/scev.o: in function `compute_max_iters':
>>> kernel/bpf/scev.c:2062:(.text+0x1193): undefined reference to `__udivdi3'
>>> ld: kernel/bpf/scev.c:2067:(.text+0x11b7): undefined reference to `__udivdi3'

Eduard,

the series is in bpf-next, so all 32-bit builds with BPF_SYSCALL=y
are broken there now. Pls send a follow up asap.
Every u64 '/' and the '%' in compute_max_iters() need div64_u64()
and div64_u64_rem().
The rest looks fine. base/step are u16 and imod() uses div_s64_rem().

Re: [PATCH bpf-next v3 34/51] bpf: use SCEV to widen bounded loops

[Eduard Zingerman]

On Sat, 2026-10-10 at 14:55 +0000, Alexei Starovoitov wrote:
> On Sat, Oct 10, 2026 at 09:25 AM kernel test robot <lkp@intel.com> wrote:
> >    ld: kernel/bpf/scev.o: in function `compute_max_iters':
> > > > kernel/bpf/scev.c:2062:(.text+0x1193): undefined reference to `__udivdi3'
> > > > ld: kernel/bpf/scev.c:2067:(.text+0x11b7): undefined reference to `__udivdi3'
> 
> Eduard,
> 
> the series is in bpf-next, so all 32-bit builds with BPF_SYSCALL=y
> are broken there now. Pls send a follow up asap.
> Every u64 '/' and the '%' in compute_max_iters() need div64_u64()
> and div64_u64_rem().
> The rest looks fine. base/step are u16 and imod() uses div_s64_rem().

Working on a followup, will send it today.

[PATCH bpf-next v3 35/51] bpf: avoid widening registers that hinder exact stack-slot tracking

[Eduard Zingerman]

For loops like this:

  int arr[10];
  for (int i = 0; i < 10; i++)
    arr[i] = i;
  use(arr[i]);

SCEV-based loop widening converts `i` from being enumerated 10 times
as a constant value 0, 1, ..., 9 to being enumerated once as a range
[0..9]. From the point of view of the main verification pass, this
replaces constant-offset stack access with varying-offset
stack access.

The verifier tracks stack values accessed through varying offsets much
less precisely than those accessed via constant offsets.
For example, the read at use(arr[i]) would yield STACK_MISC.
Precision might be lost for the following operations:
- spills (BPF_ST/STX to the stack): stored value can no longer be
  recovered when read back, e.g. after the loop;
- fills (BPF_LDX from the stack): a varying-offset read of a spilled
  pointer yields a SCALAR_VALUE, so the pointer is lost.
- calls that construct objects on the stack (dynptr, iter, irq_flag,
  res_spin_lock): the stack argument is required to have a
  constant offset.

This commit adds logic to avoid widening registers used by
such instructions:
- During the main SCEV computation stage, collect_store_base_regs():
  - inspects instructions accessing the stack (spill, fill, call to a
    function listed in bpf_needs_fixed_stack_off), and
  - if there is a SCEV expression for a stack base address register,
    records which registers the expression depends on in
    the bpf_loop->store_base_reg field.
- `store_base_reg`s accumulated for nested loops are propagated
  to outer loops as well.
- During the loop-widening stage, bpf_compute_loop_iters() refuses to
  widen a loop when some of the registers the loop changes appear in
  store_base_regs.

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 include/linux/bpf_verifier.h | 10 ++++++
 kernel/bpf/scev.c            | 86 ++++++++++++++++++++++++++++++++++++++++++--
 kernel/bpf/verifier.c        | 49 +++++++++++++++++++++++++
 3 files changed, 143 insertions(+), 2 deletions(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index d760d2283ac5..308e779c3f74 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -708,6 +708,9 @@ struct bpf_loop_exit {
 	int to; /* instruction outside the loop */
 };
 
+/* SCEV/widening register space: r0..r10 plus every stack slot of a frame. */
+#define BPF_SCEV_REGS_NUM (MAX_BPF_REG + MAX_BPF_STACK_SLOTS)
+
 struct bpf_loop {
 	struct bpf_backedge backedges[MAX_BACKEDGES];
 	/* edges exiting from this loop, includes edges from nested loops */
@@ -717,6 +720,12 @@ struct bpf_loop {
 	bool irreducible;
 	bool backedges_overflow;
 	bool exits_overflow;
+	/*
+	 * Loop-entry registers used to compute stack addresses for loads,
+	 * stores and calls requiring fixed stack offsets, in this and nested
+	 * loops.
+	 */
+	unsigned long store_base_regs[BITS_TO_LONGS(BPF_SCEV_REGS_NUM)];
 };
 
 struct bpf_insn_aux_data {
@@ -1876,6 +1885,7 @@ int bpf_stack_liveness_init(struct bpf_verifier_env *env);
 void bpf_stack_liveness_free(struct bpf_verifier_env *env);
 int bpf_live_stack_query_init(struct bpf_verifier_env *env, struct bpf_verifier_state *st);
 const unsigned long *bpf_may_write_mask(struct bpf_verifier_env *env, u32 insn_idx);
+bool bpf_needs_fixed_stack_off(struct bpf_verifier_env *env, int insn_idx);
 bool bpf_stack_slot_alive(struct bpf_verifier_env *env, u32 frameno, u32 spi);
 int bpf_compute_live_registers(struct bpf_verifier_env *env);
 
diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
index 05dd8e439609..adad32d8cb12 100644
--- a/kernel/bpf/scev.c
+++ b/kernel/bpf/scev.c
@@ -10,7 +10,7 @@
 #include <linux/tnum.h>
 #include <linux/overflow.h>
 
-#define REGS_NUM (MAX_BPF_REG + MAX_BPF_STACK_SLOTS)
+#define REGS_NUM BPF_SCEV_REGS_NUM
 #define UNKNOWN_EXPR_ID 0
 #define OPAQUE_EXPR_ID  1
 
@@ -1100,6 +1100,70 @@ static void reset_scevs_at_indirect_writes(struct bpf_verifier_env *env, struct
 	}
 }
 
+/*
+ * Collect loop-entry registers referenced by expr 'id', best effort.
+ * DEPTH_LIMIT may leave dependencies unrecorded. This mask only avoids
+ * widening that would lose stack-access precision; missing dependencies
+ * may cause false rejections, but this is not a soundness issue.
+ */
+static void or_expr_regs(struct bpf_verifier_env *env, u32 id, unsigned long *mask)
+{
+	struct scev *scev = env->scev;
+	u32 order;
+
+	scev->stack_sz = 0;
+	expr_stack_push(scev, id);
+	while (expr_next(scev, &id, &order)) {
+		if ((order & PRE) && scev->exprs[id].op == REG)
+			__set_bit(scev->exprs[id].params[0], mask);
+	}
+}
+
+/* Mask of argument registers (R1..R5) a call at 'idx' passes by register. */
+static u16 call_params_mask(struct bpf_verifier_env *env, int idx)
+{
+	struct bpf_insn *insn = &env->prog->insnsi[idx];
+	struct bpf_call_summary cs;
+	int n = bpf_get_call_summary(env, insn, &cs) ? cs.arg_slot_cnt : MAX_BPF_FUNC_REG_ARGS;
+
+	return n ? GENMASK(BPF_REG_1 + n - 1, BPF_REG_1) : 0;
+}
+
+/*
+ * For instructions like:
+ * - *(u64 *)(rBase + off) = rX
+ * - rX = *(u64 *)(rBase + off)
+ * - calls that construct objects on stack (e.g. dynptr_from_mem(rBase, ...))
+ * When 'rBase' can be a stack pointer and is derived from some registers Rs
+ * defined at loop entry, record Rs into 'mask'.
+ */
+static void collect_store_base_regs(struct bpf_verifier_env *env,
+				    struct env *cur_env, int idx, unsigned long *mask)
+{
+	struct bpf_insn_aux_data *aux = env->insn_aux_data;
+	struct bpf_insn *insn = &env->prog->insnsi[idx];
+	u8 class = BPF_CLASS(insn->code);
+	u8 size = BPF_SIZE(insn->code);
+	u16 base_regs = 0;
+	u32 r;
+
+	if (size == BPF_W || size == BPF_DW) {
+		if ((class == BPF_STX || class == BPF_ST) && insn->dst_reg != BPF_REG_FP)
+			base_regs |= BIT(insn->dst_reg);
+		else if (class == BPF_LDX && insn->src_reg != BPF_REG_FP)
+			base_regs |= BIT(insn->src_reg);
+	}
+
+	if (class == BPF_JMP && BPF_OP(insn->code) == BPF_CALL &&
+	    bpf_needs_fixed_stack_off(env, idx))
+		base_regs |= call_params_mask(env, idx);
+
+	base_regs &= aux[idx].stack_ptrs;
+	for (r = 0; r < MAX_BPF_REG; r++)
+		if (base_regs & BIT(r))
+			or_expr_regs(env, cur_env->reg2expr[r], mask);
+}
+
 /* Find the topmost loop header containing idx inside cur_header, or -1 if none. */
 static int topmost_nested_loop(struct bpf_verifier_env *env, int idx, int cur_header)
 {
@@ -1184,6 +1248,7 @@ static int compute_scev_for_loop(struct bpf_verifier_env *env, int cur_header)
 	struct bpf_iarray *succ;
 	bool log_level2 = env->log.level & BPF_LOG_LEVEL2;
 	int s, i, err, idx, succ_idx;
+	u32 r;
 
 	if (log_level2)
 		bpf_log(&env->log, "Computing SCEV for loop at %d:\n", cur_header);
@@ -1197,6 +1262,7 @@ static int compute_scev_for_loop(struct bpf_verifier_env *env, int cur_header)
 		if (!header_env)
 			return -ENOMEM;
 		/* The freshly allocated environment has all expressions unknown. */
+		bitmap_fill(cur_loop->store_base_regs, REGS_NUM);
 		header_env->empty = false;
 		return 0;
 	}
@@ -1233,6 +1299,9 @@ static int compute_scev_for_loop(struct bpf_verifier_env *env, int cur_header)
 			 */
 			if (log_level2)
 				memcpy(old_env, cur_env, sizeof(*old_env));
+			/* Pull the nested loop's stack-store base dependencies up. */
+			for_each_set_bit(r, nested_loop->store_base_regs, BPF_SCEV_REGS_NUM)
+				or_expr_regs(env, cur_env->reg2expr[r], cur_loop->store_base_regs);
 			forget_non_invariants(env, cur_env, idx);
 			if (log_level2)
 				log_env_changes(env, LOG_AT_TRANSFER, old_env, cur_env, idx);
@@ -1254,6 +1323,7 @@ static int compute_scev_for_loop(struct bpf_verifier_env *env, int cur_header)
 			for (;;) {
 				if (log_level2)
 					memcpy(old_env, cur_env, sizeof(*old_env));
+				collect_store_base_regs(env, cur_env, idx, cur_loop->store_base_regs);
 				err = transfer(env, cur_env, idx);
 				if (err)
 					goto out;
@@ -2451,6 +2521,7 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
 	for (r = 0; r < REGS_NUM; r++) {
 		struct bpf_reg_state *reg;
 		u32 ra, r_scev, r_expr;
+		bool spill_base = false;
 
 		if (!scev_reg_alive(env, st, r))
 			continue;
@@ -2472,6 +2543,11 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
 		/* If SCEV for r is (linear <reg> <slope>) */
 		if (is_simple_linear(scev, r_scev, &base_reg, &slope_imm) &&
 		    reg && is_widenable_reg_type(reg)) {
+			/* Spills at varying offsets lose precision */
+			if (test_bit(r, loop->store_base_regs)) {
+				spill_base = true;
+				goto cant_widen;
+			}
 			continue;
 		}
 
@@ -2481,15 +2557,21 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
 
 		/* (any 1 (any 2 (any 3 4))) */
 		if (is_widenable_any(env, cur_func, header_env, r, r_scev)) {
+			if (test_bit(r, loop->store_base_regs)) {
+				spill_base = true;
+				goto cant_widen;
+			}
 			continue;
 		}
-
+cant_widen:
 		if (log->level & BPF_LOG_LEVEL2) {
 			r_expr = header_env->reg2expr[r];
 			bpf_log(log, "loop header at %d, can't widen ", insn_idx);
 			log_reg(env, r);
 			bpf_log(log, ", expr is ");
 			log_expr(env, r_expr);
+			if (spill_base)
+				bpf_log(log, ", requires exact stack-offset tracking");
 			bpf_log(log, "\n");
 		}
 		return 0;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index a597a8c18972..2f190abe3448 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -14257,6 +14257,55 @@ static bool kfunc_spin_allowed(struct bpf_verifier_env *env, s32 func_id, s16 of
 	return *kfunc.flags & KF_SPINLOCK_SAFE;
 }
 
+/*
+ * True if insn calls a helper/kfunc that requires one of its arguments to
+ * be a stack pointer with a constant offset.
+ */
+bool bpf_needs_fixed_stack_off(struct bpf_verifier_env *env, int insn_idx)
+{
+	const struct bpf_insn *insn = &env->prog->insnsi[insn_idx];
+	const struct bpf_func_proto *fn;
+	struct bpf_kfunc_desc *desc;
+	u32 *flags, btf_id;
+	int i;
+
+	if (bpf_helper_call(insn)) {
+		if (bpf_get_helper_proto(env, insn->imm, &fn) < 0)
+			return false;
+	} else if (bpf_pseudo_kfunc_call(insn)) {
+		desc = find_kfunc_desc(env->prog, insn->imm, insn->off);
+		if (!desc)
+			return false;
+		fn = &desc->proto;
+	} else {
+		return false;
+	}
+
+	/* Both initialized and uninitialized stack dynptrs need a fixed offset. */
+	for (i = 0; i < ARRAY_SIZE(fn->arg_type); i++)
+		if (arg_type_is_dynptr(fn->arg_type[i]))
+			return true;
+
+	/* vmlinux kfuncs only */
+	if (!bpf_pseudo_kfunc_call(insn) || insn->off != 0)
+		return false;
+	btf_id = insn->imm;
+
+	flags = btf_kfunc_flags(btf_vmlinux, btf_id, env->prog);
+	if (flags && (*flags & (KF_ITER_NEW | KF_ITER_NEXT | KF_ITER_DESTROY)))
+		return true;
+
+	if (btf_id == special_kfunc_list[KF_bpf_res_spin_lock] ||
+	    btf_id == special_kfunc_list[KF_bpf_res_spin_unlock] ||
+	    btf_id == special_kfunc_list[KF_bpf_res_spin_lock_irqsave] ||
+	    btf_id == special_kfunc_list[KF_bpf_res_spin_unlock_irqrestore] ||
+	    btf_id == special_kfunc_list[KF_bpf_local_irq_save] ||
+	    btf_id == special_kfunc_list[KF_bpf_local_irq_restore])
+		return true;
+
+	return false;
+}
+
 static bool is_sync_callback_calling_kfunc(u32 btf_id)
 {
 	return is_bpf_rbtree_add_kfunc(btf_id);

-- 
2.53.0

Re: [PATCH bpf-next v3 35/51] bpf: avoid widening registers that hinder exact stack-slot tracking

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 1824 bytes --]

> bpf: avoid widening registers that hinder exact stack-slot tracking

TL;DR: Nit only: BPF_SCEV_REGS_NUM in bpf_verifier.h appears to be defined
twice, identically, with a duplicate comment; it compiles, but the copies
could diverge, so the new one could be dropped.

> diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
> --- a/include/linux/bpf_verifier.h
> +++ b/include/linux/bpf_verifier.h

[ ... ]

> @@ -708,6 +708,9 @@ struct bpf_loop_exit {
>  	int to; /* instruction outside the loop */
>  };
>
> +/* SCEV/widening register space: r0..r10 plus every stack slot of a frame. */
> +#define BPF_SCEV_REGS_NUM (MAX_BPF_REG + MAX_BPF_STACK_SLOTS)
> +
>  struct bpf_loop {
>  	struct bpf_backedge backedges[MAX_BACKEDGES];
>  	/* edges exiting from this loop, includes edges from nested loops */

Isn't this a second definition of BPF_SCEV_REGS_NUM, comment included?

The same header already has it, added earlier in the series by "bpf: use
SCEV to widen bounded loops" for struct bpf_loop_iters::precise_regs:

  /* SCEV/widening register space: r0..r10 plus every stack slot of a frame. */
  #define BPF_SCEV_REGS_NUM (MAX_BPF_REG + MAX_BPF_STACK_SLOTS)

That earlier definition sits above struct bpf_loop, so
store_base_regs[BITS_TO_LONGS(BPF_SCEV_REGS_NUM)] can use it directly.  The
two definitions are token-identical, so it compiles cleanly, but a later
edit to one copy could silently diverge from the other.

Could the new #define and its comment be dropped from this hunk?  The
duplicate is still present at the end of the series (726d0559f576).


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/37923187511

Re: [PATCH bpf-next v3 35/51] bpf: avoid widening registers that hinder exact stack-slot tracking

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] bpf: call_params_mask clobbers callee-saved registers for kfuncs with >5 args
- [Medium] bpf: bpf_needs_fixed_stack_off ignores module kfuncs and stack dynptrs
--

commit c21aed6872d1b94060c148c7501830216020d7e4
Author: Eduard Zingerman <eddyz87@gmail.com>

bpf: avoid widening registers that hinder exact stack-slot tracking

This commit adds logic to avoid widening registers used by stack-accessing
instructions during SCEV-based loop analysis. This prevents replacing
constant-offset stack accesses with varying-offset accesses, preserving
verifier precision for spills, fills, and calls requiring fixed stack
offsets.

> diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
> index 05dd8e439609b..adad32d8cb124 100644
> --- a/kernel/bpf/scev.c
> +++ b/kernel/bpf/scev.c
[ ... ]
> @@ -1100,6 +1100,70 @@ static void reset_scevs_at_indirect_writes(struct bpf_verifier_env *env, struct
[ ... ]
> +/* Mask of argument registers (R1..R5) a call at 'idx' passes by register. */
> +static u16 call_params_mask(struct bpf_verifier_env *env, int idx)
> +{
> +	struct bpf_insn *insn = &env->prog->insnsi[idx];
> +	struct bpf_call_summary cs;
> +	int n = bpf_get_call_summary(env, insn, &cs) ? cs.arg_slot_cnt : MAX_BPF_FUNC_REG_ARGS;
> +
> +	return n ? GENMASK(BPF_REG_1 + n - 1, BPF_REG_1) : 0;
> +}

[Severity: Medium]
Does this code overflow the bitmask into callee-saved registers when a
kfunc takes arguments on the stack?

bpf_get_call_summary() returns the total number of arguments in
cs.arg_slot_cnt, which can exceed 5 (MAX_BPF_FUNC_REG_ARGS) for kfuncs
that accept stack arguments.

Since GENMASK(BPF_REG_1 + n - 1, BPF_REG_1) does not cap 'n', an 'n'
greater than 5 will set bits for callee-saved registers (R6-R10).

If one of these unrelated registers happens to hold a stack pointer,
collect_store_base_regs() might erroneously record it as a base register
for the call. This could falsely prevent SCEV from widening unrelated
loops, causing the verifier to unnecessarily unroll the loop and
potentially hit complexity limits.

> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index a597a8c189723..2f190abe34489 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
[ ... ]
> @@ -14257,6 +14257,55 @@ static bool kfunc_spin_allowed(struct bpf_verifier_env *env, s32 func_id, s16 of
[ ... ]
> +bool bpf_needs_fixed_stack_off(struct bpf_verifier_env *env, int insn_idx)
> +{
> +	const struct bpf_insn *insn = &env->prog->insnsi[insn_idx];
[ ... ]
> +	/* Both initialized and uninitialized stack dynptrs need a fixed offset. */
> +	for (i = 0; i < ARRAY_SIZE(fn->arg_type); i++)
> +		if (arg_type_is_dynptr(fn->arg_type[i]))
> +			return true;

[Severity: Medium]
Does this loop miss dynptrs that are passed on the stack?

Because fn->arg_type is strictly bounded to 5 elements, this iteration
will skip checking a dynptr if it is passed as a 6th argument (or later)
on the stack.

If missed, collect_store_base_regs() fails to record the base registers,
allowing the SCEV pass to widen the stack pointer and turn its constant
offset into a variable range. The verifier will then reject the dynptr
call because it strictly enforces a constant offset.

> +
> +	/* vmlinux kfuncs only */
> +	if (!bpf_pseudo_kfunc_call(insn) || insn->off != 0)
> +		return false;

[Severity: Medium]
Will this check skip BPF module iterators?

By returning false if insn->off != 0, this explicitly ignores any kfuncs
exposed by BPF modules. Modules can provide their own iterators (like
bpf_iter_testmod_seq_new with KF_ITER_NEW).

Similar to the dynptr issue above, skipping these iterators means the
base registers won't be recorded. This causes the stack pointer to be
widened and leads to valid programs utilizing module iterators being
falsely rejected by the verifier.

> +	btf_id = insn->imm;
> +
> +	flags = btf_kfunc_flags(btf_vmlinux, btf_id, env->prog);
> +	if (flags && (*flags & (KF_ITER_NEW | KF_ITER_NEXT | KF_ITER_DESTROY)))
> +		return true;

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261009-scev-minimal-rebase-v3-0-bcba9bc53ac9@gmail.com?part=35

[PATCH bpf-next v3 36/51] bpf: bpf_func_state size optimization

[Eduard Zingerman]

Pack bpf_reg_state to bring it's size from 88 bytes to 80.
Consequently, bpf_func_state changes from 1048 to 960.
Thus bpf_func_state moves from 2K to 1K allocator bucket.
1K bucket is the one it fit before the SCEV series.

With this optimization SCEV increases memory consumption on a subset
of Meta internal programs by 3%, without this optimization the
increase is 48%.

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 include/linux/bpf_verifier.h | 4 ++--
 kernel/bpf/states.c          | 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 308e779c3f74..54cb6d3d0653 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -184,9 +184,9 @@ struct bpf_reg_state {
 	 * other registers. Kept outside the metadata union for ID remapping
 	 * during state comparisons.
 	 */
-	u32 map_uid;
+	u32 map_uid:31;
 	/* if (!precise && SCALAR_VALUE) min/max/tnum don't affect safety */
-	bool precise;
+	u32 precise:1;
 };
 
 static inline s64 reg_smin(const struct bpf_reg_state *reg)
diff --git a/kernel/bpf/states.c b/kernel/bpf/states.c
index c62ea992967b..953700822fa7 100644
--- a/kernel/bpf/states.c
+++ b/kernel/bpf/states.c
@@ -1171,7 +1171,7 @@ static bool states_maybe_looping(struct bpf_verifier_state *old,
 	fcur = cur->frame[fr];
 	for (i = 0; i < MAX_BPF_REG; i++)
 		if (memcmp(&fold->regs[i], &fcur->regs[i],
-			   offsetof(struct bpf_reg_state, precise)))
+			   offsetofend(struct bpf_reg_state, step)))
 			return false;
 	return true;
 }

-- 
2.53.0

Re: [PATCH bpf-next v3 36/51] bpf: bpf_func_state size optimization

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 4096 bytes --]

> bpf: bpf_func_state size optimization

TL;DR: The narrower memcmp prefix in states_maybe_looping() may stop comparing
id, parent_id and map_uid, so a terminating program accepted before could be
wrongly rejected as an "infinite loop"; the commit message doesn't mention it.

> diff --git a/kernel/bpf/states.c b/kernel/bpf/states.c
> index c62ea992967b..953700822fa7 100644
> --- a/kernel/bpf/states.c
> +++ b/kernel/bpf/states.c

[ ... ]

> @@ -1171,7 +1171,7 @@ static bool states_maybe_looping(struct bpf_verifier_state *old,
>  	fcur = cur->frame[fr];
>  	for (i = 0; i < MAX_BPF_REG; i++)
>  		if (memcmp(&fold->regs[i], &fcur->regs[i],
> -			   offsetof(struct bpf_reg_state, precise)))
> +			   offsetofend(struct bpf_reg_state, step)))
>  			return false;
>  	return true;
>  }

Does this narrower prefix drop id, parent_id and map_uid from the
comparison?

The old length, offsetof(struct bpf_reg_state, precise), was 80 bytes and
covered the raw id, parent_id and map_uid.  The new
offsetofend(struct bpf_reg_state, step) is 68 bytes, and id, parent_id and
map_uid are all declared after step in the struct, so none of them are
compared any more.  offsetof() cannot name the precise bitfield, but
offsetofend(struct bpf_reg_state, parent_id) would have kept id and
parent_id in the comparison.

Commit 8901cee9316d said this prefix comparison "continues to cover the
complete value state", and this commit message does not mention that the
comparison is being narrowed.

Before this change, the current frame's registers had to carry identical
raw ids before infinite-loop detection could fire.  Now the decision rests
on states_equal(env, &sl->state, cur, EXACT).  In EXACT mode regs_exact()
compares ids only with plain check_ids():

    return memcmp(rold, rcur, offsetof(struct bpf_reg_state, id)) == 0 &&
           check_ids(rold->id, rcur->id, idmap) && ...

That skips the extra checks that regsafe()/check_scalar_ids() do for
scalars: the ADD_CONST flags must match, and the base id (with the flag
stripped) must map consistently.  The comment above check_scalar_ids()
describes this exact gap: "Without the base check, idmap gets two
independent entries A->B and A|flag->C|flag, missing that A->C conflicts
with A->B."

Can a program that terminates, and was accepted before, now be rejected
with "infinite loop detected"?  For example (privileged, so ADD_CONST
linking is active):

  r6 = rand&7; r7 = r6          (r6,r7 id A)
  r8 = rand&7; r9 = r8; r9 += 1 (r8 id C, r9 id C|ADD64, delta 1)
 L: if r6 > 3 goto out
    if r9 <= 4 goto out          (on entry r9 is tied to r8, so both
                                  branches are possible)
    r6 = rand&7; r7 = r6; r9 = r6; r9 += 1
                                 (r9 is now tied to r6: id B|ADD64)
    r8 = rand&7                  (id D)
    goto L
 out: use r6..r9; exit

On the second visit to L the register values match the checkpoint exactly,
and only the ids differ (A->B, C|ADD->B|ADD, C->D).  check_ids() accepts
this mapping even though r9 now depends on r6 instead of r8.

The old raw-id prefilter rejected the match (r6 id A != B), so exploration
went on: r6 <= 3 forces r9 <= 4 through sync_linked_regs(), every path from
the second visit reaches exit, and the program was accepted.

With this patch, states_maybe_looping() returns true, states_equal(EXACT)
succeeds, and bpf_is_state_visited() returns -EINVAL.

The same weakness could already be reached before this patch through ids
in stack spills or caller frames, which the prefilter never covered.  This
patch extends it to registers in the current frame.

Should this compare up to offsetofend(struct bpf_reg_state, parent_id) and
compare map_uid separately?  Alternatively, if dropping the ids is
intended, could the commit message say so, and could regs_exact() check
ids the way check_scalar_ids() does?


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/37923187511

[PATCH bpf-next v3 37/51] selftests/bpf: __msg_next tag for matching messages on consecutive lines

[Eduard Zingerman]

Same as __msg, but expects the match on the line immediately after
the last match, not just any later line.
E.g.:

  __msg_next("a")
  __msg_next("c")

This would match consecutive output "a\n" "c\n",
but would not match output "a\n" "b\n" "c\n".

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 tools/testing/selftests/bpf/progs/bpf_misc.h |  6 ++++++
 tools/testing/selftests/bpf/test_loader.c    | 11 +++++++++++
 2 files changed, 17 insertions(+)

diff --git a/tools/testing/selftests/bpf/progs/bpf_misc.h b/tools/testing/selftests/bpf/progs/bpf_misc.h
index f3dbc3b59bff..d7f706c3bc4a 100644
--- a/tools/testing/selftests/bpf/progs/bpf_misc.h
+++ b/tools/testing/selftests/bpf/progs/bpf_misc.h
@@ -43,6 +43,10 @@
  * __msg_unpriv      Same as __msg but for unprivileged mode.
  * __not_msg_unpriv  Same as __not_msg but for unprivileged mode.
  *
+ * __msg_next        Same as __msg but expects the match to be on the next
+ * __msg_next_unpriv log line compared to the last match, not just some line
+ *                   after the last match.
+ *
  * __stderr          Message expected to be found in bpf stderr stream. The
  *                   same regex rules apply like __msg.
  * __stderr_unpriv   Same as __stderr but for unpriveleged mode.
@@ -145,6 +149,7 @@
 
 #define __msg(msg)		__test_tag("test_expect_msg=" msg)
 #define __not_msg(msg)		__test_tag("test_expect_not_msg=" msg)
+#define __msg_next(msg)		__test_tag("test_expect_msg_next=" msg)
 #define __xlated(msg)		__test_tag("test_expect_xlated=" msg)
 #define __jited(msg)		__test_tag("test_jited=" msg)
 #define __failure		__test_tag("test_expect_failure")
@@ -153,6 +158,7 @@
 #define __skip(reason)		__test_tag("test_skip=" reason)
 #define __msg_unpriv(msg)	__test_tag("test_expect_msg_unpriv=" msg)
 #define __not_msg_unpriv(msg)	__test_tag("test_expect_not_msg_unpriv=" msg)
+#define __msg_next_unpriv(msg)	__test_tag("test_expect_msg_next_unpriv=" msg)
 #define __xlated_unpriv(msg)	__test_tag("test_expect_xlated_unpriv=" msg)
 #define __jited_unpriv(msg)	__test_tag("test_jited_unpriv=" msg)
 #define __failure_unpriv	__test_tag("test_expect_failure_unpriv")
diff --git a/tools/testing/selftests/bpf/test_loader.c b/tools/testing/selftests/bpf/test_loader.c
index a89890cd56d8..18d47e925400 100644
--- a/tools/testing/selftests/bpf/test_loader.c
+++ b/tools/testing/selftests/bpf/test_loader.c
@@ -508,6 +508,16 @@ static int parse_test_spec(struct test_loader *tester,
 			if (err)
 				goto cleanup;
 			spec->mode_mask |= UNPRIV;
+		} else if ((msg = str_has_pfx(s, "test_expect_msg_next="))) {
+			err = __push_msg(msg, true, false, &spec->priv.expect_msgs);
+			if (err)
+				goto cleanup;
+			spec->mode_mask |= PRIV;
+		} else if ((msg = str_has_pfx(s, "test_expect_msg_next_unpriv="))) {
+			err = __push_msg(msg, true, false, &spec->unpriv.expect_msgs);
+			if (err)
+				goto cleanup;
+			spec->mode_mask |= UNPRIV;
 		} else if ((msg = str_has_pfx(s, "test_jited="))) {
 			if (arch_mask == 0) {
 				PRINT_FAIL("__jited used before __arch_*");
@@ -998,6 +1008,7 @@ void validate_msgs(const char *log_buf, struct expected_msgs *msgs,
 
 		no_match   = !msg->negative && !match->start;
 		wrong_line = !msg->negative &&
+			     match->start &&
 			     msg->on_next_line &&
 			     prev_match && prev_match->line + 1 != match->line;
 		unexpected = msg->negative && match->start;

-- 
2.53.0

[PATCH bpf-next v3 38/51] selftests/bpf: bound UNIX socket path loops by sun_path size

[Eduard Zingerman]

The bpf_iter/unix and skc_to_unix_sock tests index sun_path but bound
that index by sizeof(struct sockaddr_un). This allows indices 108 and
109 into the 108-byte array. With SCEV loop widening, the variable-offset
BTF access check rejects the resulting range.

Bound the index by sizeof(sun_path) in both tests. Runtime behavior for
valid UNIX addresses is unchanged.

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 tools/testing/selftests/bpf/progs/bpf_iter_unix.c         | 2 +-
 tools/testing/selftests/bpf/progs/test_skc_to_unix_sock.c | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/tools/testing/selftests/bpf/progs/bpf_iter_unix.c b/tools/testing/selftests/bpf/progs/bpf_iter_unix.c
index df3d971b21e8..3061b148f338 100644
--- a/tools/testing/selftests/bpf/progs/bpf_iter_unix.c
+++ b/tools/testing/selftests/bpf/progs/bpf_iter_unix.c
@@ -70,7 +70,7 @@ int dump_unix(struct bpf_iter__unix *ctx)
 
 			for (i = 1; i < len; i++) {
 				/* unix_validate_addr() tests this upper bound. */
-				if (i >= sizeof(struct sockaddr_un))
+				if (i >= sizeof(unix_sk->addr->name->sun_path))
 					break;
 
 				BPF_SEQ_PRINTF(seq, "%c",
diff --git a/tools/testing/selftests/bpf/progs/test_skc_to_unix_sock.c b/tools/testing/selftests/bpf/progs/test_skc_to_unix_sock.c
index 4cfa42aa9436..c5c643064eb5 100644
--- a/tools/testing/selftests/bpf/progs/test_skc_to_unix_sock.c
+++ b/tools/testing/selftests/bpf/progs/test_skc_to_unix_sock.c
@@ -29,7 +29,7 @@ int BPF_PROG(unix_listen, struct socket *sock, int backlog)
 	len = unix_sk->addr->len - sizeof(short);
 	path[0] = '@';
 	for (i = 1; i < len; i++) {
-		if (i >= (int)sizeof(struct sockaddr_un))
+		if (i >= (int)sizeof(unix_sk->addr->name->sun_path))
 			break;
 
 		path[i] = unix_sk->addr->name->sun_path[i];

-- 
2.53.0

[PATCH bpf-next v3 39/51] selftests/bpf: test for stack-pointer subrange pruning

[Eduard Zingerman]

Validate changes from:
"bpf: allow subrange relations for PTR_TO_STACK in regsafe()".
Add a test case that checks that a widened stack pointer with range
[-72, -16] is considered a superset of a stack pointer with range
[-64, -16].

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 .../selftests/bpf/progs/verifier_stack_ptr.c       | 36 ++++++++++++++++++++++
 1 file changed, 36 insertions(+)

diff --git a/tools/testing/selftests/bpf/progs/verifier_stack_ptr.c b/tools/testing/selftests/bpf/progs/verifier_stack_ptr.c
index 3e0bea9819ca..3e9d8fc606ae 100644
--- a/tools/testing/selftests/bpf/progs/verifier_stack_ptr.c
+++ b/tools/testing/selftests/bpf/progs/verifier_stack_ptr.c
@@ -586,4 +586,40 @@ __naked void stack_check_size_512_with_may_goto(void)
 }
 #endif
 
+/*
+ * Verify that old PTR_TO_STACK state is considered a super-set of
+ * new PTR_TO_STACK state when new variable range is a sub-range
+ * of the old range, e.g. old [-72, -16] vs new [-64, -16].
+ */
+SEC("socket")
+__success
+__log_level(2)
+__flag(BPF_F_TEST_STATE_FREQ)
+/*
+ * r7 is widened to [-72, -16] at the loop header (insn 3),
+ * the loop body sees [-64, -8] after 'r7 += 8'
+ */
+__msg("loop header at 3, widening r7 to -72..-16 step 8")
+__msg("R7=fp(smin=smin32=-64,smax=smax32=-8")
+/*
+ * back-edge state is clamped to the remaining iterations and pruned
+ * at the header, because [-64, -16] is within the widened [-72, -16]
+ */
+__msg("loop header at 3, clamping r7 to -64..-16 step 8")
+__msg("from 5 to 3: safe")
+__msg("processed 9 insns")
+__naked void stack_ptr_subrange_in_loop(void)
+{
+	asm volatile ("					\
+	r7 = r10;					\
+	r7 += -72;					\
+	r6 = 0;						\
+	r7 += 8;					\
+	r6 += 1;					\
+	if r6 < 8 goto -3;				\
+	r0 = 0;						\
+	exit;						\
+"	::: __clobber_all);
+}
+
 char _license[] SEC("license") = "GPL";

-- 
2.53.0

[PATCH bpf-next v3 40/51] selftests/bpf: tests for may_write stack-liveness tracking

[Eduard Zingerman]

Branches to cover:
  - per-insn writes (record_stack_access_off):
      * off_cnt==1, fully covered slots   -> def + may_def (equal)
      * partial coverage                  -> may_def only (no def)
      * off_cnt 2-4, precise offsets      -> may_def only, per offset
  - whole-frame writes:
      * off_cnt==0 (offset lost)          -> may_def SPIS_ALL
      * ARG_IMPRECISE (frame lost)        -> may_def SPIS_ALL per frame
  - merge_instances():
      * both passes touch a frame         -> may_def union
      * one pass leaves a frame untouched -> may_def preserved
  - merge_may_write(): a callee's writes into ancestor frames are
    summarized as may_def at the caller's call instruction
  - analyze_subprog()                     -> may_def stack slot range
                                             when callback is unknown.

Tests exercising them:
  - may_write_two_precise_offsets  - off_cnt 2-4 precise multi-offset
  - imprecise_frame_write          - cross-frame ARG_IMPRECISE write,
                                     plus its caller-side summarization
  - merge_preserves_may_write      - map-value vs stack pointer at two
                                     callsites: union is preserved when
                                     one pass does not touch the frame
  - must_write_not_same_slot       - off_cnt==0 whole-frame may_def
  - two_byte_write_no_kill         - partial write, may_def without def
  - kfunc_iter_stack_liveness      - full-slot def == may_def
  - helper_output_unknown_size     - output size unknown to liveness,
                                     may_def without use or def
  - caller_stack_write,
    conditional_stx_in_subprog,
    shared_instance_must_write_overwrite
                                   - callee writes summarized as may_def
                                     at the call site
  - bpf_loop_two_callbacks         - call to unknown callback marks full
                                     stack range as may_def.

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 .../selftests/bpf/progs/verifier_live_stack.c      | 174 ++++++++++++++++++++-
 1 file changed, 170 insertions(+), 4 deletions(-)

diff --git a/tools/testing/selftests/bpf/progs/verifier_live_stack.c b/tools/testing/selftests/bpf/progs/verifier_live_stack.c
index 6e8f50618580..0b89c5776824 100644
--- a/tools/testing/selftests/bpf/progs/verifier_live_stack.c
+++ b/tools/testing/selftests/bpf/progs/verifier_live_stack.c
@@ -83,6 +83,31 @@ __naked void must_write_not_same_slot(void)
 	: __clobber_all);
 }
 
+SEC("socket")
+__log_level(2)
+__msg("stack use/def subprog#0 may_write_two_precise_offsets (d0,cs0):")
+/*
+ * r1 is fp-8 or fp-16, with both offsets tracked precisely (off_cnt == 2).
+ * A write through it cannot set def, but marks both candidate slots as may_def.
+ */
+__msg("6: (7b) *(u64 *)(r1 +0) = r0         ; may_def: fp0-8 fp0-16")
+__naked void may_write_two_precise_offsets(void)
+{
+	asm volatile (
+	"call %[bpf_get_prandom_u32];"
+	"r1 = r10;"
+	"if r0 > 42 goto 1f;"
+	"r1 += -8;"
+	"goto 2f;"
+"1:"
+	"r1 += -16;"
+"2:"
+	"*(u64 *)(r1 + 0) = r0;"
+	"exit;"
+	:: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
 SEC("socket")
 __log_level(2)
 __msg("0: (7a) *(u64 *)(r10 -8) = 0         ; def: fp0-8")
@@ -114,7 +139,7 @@ __log_level(2)
 __msg("stack use/def subprog#0 caller_stack_write (d0,cs0):")
 __msg("2: (85) call pc+1                    ; may_def: fp0-8")
 __msg("stack use/def subprog#1 write_first_param (d1,cs2):")
-__msg("4: (7a) *(u64 *)(r1 +0) = 7          ; def: fp0-8")
+__msg("4: (7a) *(u64 *)(r1 +0) = 7          ; def: fp0-8 may_def: fp0-8")
 __naked void caller_stack_write(void)
 {
 	asm volatile (
@@ -134,6 +159,49 @@ static __used __naked void write_first_param(void)
 	::: __clobber_all);
 }
 
+/*
+ * Cross-frame imprecise write: imprecise_frame_writer() receives a pointer
+ * into the caller's frame (frame 0) but conditionally replaces it with a
+ * pointer into its own frame (frame 1). At the store the two are joined into
+ * ARG_IMPRECISE (frame is unknown, mask = {0,1}), so the offset is dropped and
+ * the whole of both candidate frames is conservatively marked as may_def.
+ */
+SEC("socket")
+__log_level(2)
+/* The callee's write into the caller frame is summarized at the call site. */
+__msg("stack use/def subprog#0 imprecise_frame_write (d0,cs0):")
+__msg("2: (85) call pc+2                    ; may_def: fp0-8..-{{(512|2048)}}")
+__msg("stack use/def subprog#1 imprecise_frame_writer (d1,cs2):")
+__msg("12: (7b) *(u64 *)(r1 +0) = r2         ; may_def: fp1-8..-{{(512|2048)}} fp0-8..-{{(512|2048)}}")
+__naked void imprecise_frame_write(void)
+{
+	asm volatile (
+	"r1 = r10;"
+	"r1 += -8;"
+	"call imprecise_frame_writer;"
+	"r0 = 0;"
+	"exit;"
+	::: __clobber_all);
+}
+
+static __used __naked void imprecise_frame_writer(void)
+{
+	asm volatile (
+	"r6 = r1;"			/* save arg: pointer into frame 0 */
+	"call %[bpf_get_prandom_u32];"
+	"r1 = r6;"			/* r1 = frame0-8 (arg) */
+	"if r0 == 0 goto 1f;"
+	"r1 = r10;"
+	"r1 += -8;"			/* r1 = frame1-8 (own frame) */
+"1:"
+	"r2 = 0;"
+	"*(u64 *)(r1 + 0) = r2;"	/* join frame0-8 | frame1-8 -> imprecise */
+	"r0 = 0;"
+	"exit;"
+	:: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
 SEC("socket")
 __log_level(2)
 __msg("stack use/def subprog#0 caller_stack_read (d0,cs0):")
@@ -1011,7 +1079,7 @@ __naked void four_byte_read_upper_half(void)
  */
 SEC("socket")
 __log_level(2)
-__msg("0: (7a) *(u64 *)(r10 -8) = 0         ; def: fp0-8")
+__msg("0: (7a) *(u64 *)(r10 -8) = 0         ; def: fp0-8 may_def: fp0-8")
 /* 2-byte write only partially covers the upper half: may_def, but no def. */
 __msg("1: (6a) *(u16 *)(r10 -4) = 0         ; may_def: fp0-4h")
 __msg("2: (61) r0 = *(u32 *)(r10 -4)        ; use: fp0-4h")
@@ -1364,7 +1432,7 @@ __log_level(2)
  * fp-8 live at call: callee conditionally writes it, so the slot is not killed
  * (no def), but the conditional write surfaces as may_def at the call site.
  */
-__msg("1: (7b) *(u64 *)(r10 -8) = r1        ; def: fp0-8")
+__msg("1: (7b) *(u64 *)(r10 -8) = r1        ; def: fp0-8 may_def: fp0-8")
 __msg("4: (85) call pc+2                    ; may_def: fp0-8")
 __msg("5: (79) r0 = *(u64 *)(r10 -8)        ; use: fp0-8")
 __naked void conditional_stx_in_subprog(void)
@@ -2275,7 +2343,11 @@ static __used __naked void merge_leaf_read(void)
 SEC("socket")
 __log_level(2)
 __success
-__msg("call bpf_loop#181            ; use: fp2-8..-{{(512|2048)}} fp1-8..-{{(512|2048)}} fp0-8..-{{(512|2048)}}")
+__msg("stack use/def subprog#0 bpf_loop_two_callbacks (d0,cs0):")
+__msg("5: (85) call pc+{{.*}};{{.*}} may_def: fp0-8..-{{(512|2048)}}{{$}}")
+__msg("8: (85) call pc+{{.*}};{{.*}} may_def: fp0-8..-{{(512|2048)}}{{$}}")
+__msg("call bpf_loop#181            ; use: fp2-8..-{{(512|2048)}} fp1-8..-{{(512|2048)}} fp0-8..-{{(512|2048)}}"
+      " may_def: fp2-8..-{{(512|2048)}} fp1-8..-{{(512|2048)}} fp0-8..-{{(512|2048)}}{{$}}")
 __naked void bpf_loop_two_callbacks(void)
 {
 	asm volatile (
@@ -2393,6 +2465,20 @@ __flag(BPF_F_TEST_STATE_FREQ)
 __msg("subprog#2 write_first_read_second:")
 __msg("17: (7a) *(u64 *)(r1 +0) = 42{{$}}")
 __msg("18: (79) r0 = *(u64 *)(r2 +0) // r1=fp0-8 r2=fp0-16{{$}}")
+/*
+ * The callee's write into the caller frame is summarized as may_def at the
+ * call sites, transitively across both frames (depth 2 -> 1 -> 0). The two
+ * call sites differ because the shared write_first_read_second instance
+ * accumulates the cross-pass union ({fp-8, fp-16}) at different points
+ * relative to each forwarding_rw's summarization.
+ */
+__msg("stack use/def subprog#0 shared_instance_must_write_overwrite (d0,cs0):")
+__msg("7: (85) call pc+7                    ; use: fp0-8 fp0-16 may_def: fp0-8 fp0-16")
+__msg("12: (85) call pc+2                    ; use: fp0-8 may_def: fp0-16")
+__msg("stack use/def subprog#1 forwarding_rw (d1,cs7):")
+__msg("15: (85) call pc+1                    ; use: fp0-8 fp0-16 may_def: fp0-8 fp0-16")
+__msg("stack use/def subprog#1 forwarding_rw (d1,cs12):")
+__msg("15: (85) call pc+1                    ; use: fp0-8 may_def: fp0-16")
 __msg("stack use/def subprog#2 write_first_read_second (d2,cs15):")
 /*
  * Shared across two callsites with swapped args (r1 is fp-8 on one pass,
@@ -2441,6 +2527,65 @@ static __used __naked void write_first_read_second(void)
 	::: __clobber_all);
 }
 
+/*
+ * merge_instances() must preserve may_write when one pass doesn't touch a frame.
+ * mvs_leaf is reached through a single callsite (mvs_mid) from two outer chains,
+ * so it is one shared (depth,callsite) instance analyzed twice and merged:
+ * - chain A passes r2 = map value, mvs_leaf's doesn't touch stack;
+ * - chain B passes r2 = caller stack slot fp-8.
+ * The merge must keep the stack pass's may_write even though the map pass left
+ * frame 0 untouched.
+ */
+SEC("socket")
+__success
+__log_level(2)
+__msg("stack use/def subprog#2 mvs_leaf (d2,cs19):")
+__msg("21: (7a) *(u64 *)(r2 +0) = 42         ; may_def: fp0-8")
+__naked void merge_preserves_may_write(void)
+{
+	asm volatile (
+	"r1 = 0;"
+	"*(u64 *)(r10 - 24) = r1;"		/* map key */
+	"r1 = %[map] ll;"
+	"r2 = r10;"
+	"r2 += -24;"
+	"call %[bpf_map_lookup_elem];"
+	"if r0 == 0 goto 1f;"
+	/* chain A: r2 = map value (ARG_NONE, not stack) */
+	"r1 = r10;"
+	"r1 += -16;"				/* unused fp anchor */
+	"r2 = r0;"
+	"call mvs_mid;"
+	/* chain B: r2 = caller stack slot fp-8 */
+	"r1 = r10;"
+	"r1 += -16;"
+	"r2 = r10;"
+	"r2 += -8;"
+	"call mvs_mid;"
+"1:"
+	"r0 = 0;"
+	"exit;"
+	: : __imm(bpf_map_lookup_elem), __imm_addr(map)
+	: __clobber_all);
+}
+
+static __used __naked void mvs_mid(void)
+{
+	asm volatile (
+	"call mvs_leaf;"
+	"exit;"
+	::: __clobber_all);
+}
+
+static __used __naked void mvs_leaf(void)
+{
+	asm volatile (
+	"*(u64 *)(r2 + 0) = 42;"
+	"r0 = 0;"
+	"exit;"
+	::: __clobber_all);
+}
+
 /*
  * Shared must_write when (callsite, depth) instance is reused.
  * Main calls fwd_to_stale_wr at two sites. fwd_to_stale_wr calls
@@ -2949,3 +3094,24 @@ __naked void spill_below_512_stays_precise(void)
 	"exit;"
 	::: __clobber_all);
 }
+
+/* Unknown-size outputs still carry a possible write, without a read or kill. */
+SEC("socket")
+__success __log_level(2)
+__msg("call bpf_probe_read_kernel{{.*}}; may_def: fp0-8{{$}}")
+__naked void helper_output_unknown_size(void)
+{
+	asm volatile (
+	"call %[bpf_get_prandom_u32];"
+	"r0 &= 7;"
+	"r0 += 1;"
+	"r2 = r0;"
+	"r1 = r10;"
+	"r1 += -8;"
+	"r3 = 0;"
+	"call %[bpf_probe_read_kernel];"
+	"r0 = 0;"
+	"exit;"
+	:: __imm(bpf_get_prandom_u32), __imm(bpf_probe_read_kernel)
+	: __clobber_all);
+}

-- 
2.53.0

[PATCH bpf-next v3 41/51] selftests/bpf: tests for may_def marks of atomic RMW operations

[Eduard Zingerman]

Liveness analysis should produce may_def marks for read-modify-write
atomic operations:
- lock *(u64 *)(r10 -8) += r1
- r1 = atomic64_fetch_add((u64 *)(r10 -8), r1)
- r1 = atomic64_xchg((u64 *)(r10 -8), r1)
- r0 = atomic64_cmpxchg((u64 *)(r10 -8), r0, r1)

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 .../selftests/bpf/progs/verifier_live_stack.c      | 26 ++++++++++++++++++++++
 1 file changed, 26 insertions(+)

diff --git a/tools/testing/selftests/bpf/progs/verifier_live_stack.c b/tools/testing/selftests/bpf/progs/verifier_live_stack.c
index 0b89c5776824..43f616ea90a9 100644
--- a/tools/testing/selftests/bpf/progs/verifier_live_stack.c
+++ b/tools/testing/selftests/bpf/progs/verifier_live_stack.c
@@ -2670,6 +2670,32 @@ __naked void load_acquire_dont_clear_dst(void)
 
 #endif /* CAN_USE_LOAD_ACQ_STORE_REL */
 
+SEC("socket")
+__log_level(2)
+__success
+__msg("lock *(u64 *)(r10 -8) += r1{{.*}}use: fp0-8{{.*}}may_def: fp0-8")
+__msg("r1 = atomic64_fetch_add((u64 *)(r10 -8), r1){{.*}}use: fp0-8{{.*}}may_def: fp0-8")
+__msg("r1 = atomic64_xchg((u64 *)(r10 -8), r1){{.*}}use: fp0-8{{.*}}may_def: fp0-8")
+__msg("r0 = atomic64_cmpxchg((u64 *)(r10 -8), r0, r1){{.*}}use: fp0-8{{.*}}may_def: fp0-8")
+__naked void atomic_rmw(void)
+{
+	asm volatile (
+	"r1 = 0;"
+	"*(u64 *)(r10 - 8) = r1;"
+	".8byte %[atomic_add];"
+	".8byte %[atomic_fetch_add];"
+	".8byte %[atomic_xchg];"
+	"r0 = 0;"
+	".8byte %[atomic_cmpxchg];"
+	"exit;"
+	:
+	: __imm_insn(atomic_add, BPF_ATOMIC_OP(BPF_DW, BPF_ADD, BPF_REG_10, BPF_REG_1, -8)),
+	  __imm_insn(atomic_fetch_add, BPF_ATOMIC_OP(BPF_DW, BPF_ADD | BPF_FETCH, BPF_REG_10, BPF_REG_1, -8)),
+	  __imm_insn(atomic_xchg, BPF_ATOMIC_OP(BPF_DW, BPF_XCHG, BPF_REG_10, BPF_REG_1, -8)),
+	  __imm_insn(atomic_cmpxchg, BPF_ATOMIC_OP(BPF_DW, BPF_CMPXCHG, BPF_REG_10, BPF_REG_1, -8))
+	: __clobber_all);
+}
+
 SEC("socket")
 __success
 __naked void imprecise_fill_loses_cross_frame(void)

-- 
2.53.0

[PATCH bpf-next v3 42/51] selftests/bpf: tests for stack pointer spills/fills through atomics

[Eduard Zingerman]

Check stack write tracking for atomic operations:
- Restored by 64-bit load-acquire, xchg, fetch_add and cmpxchg.
- Spilled by store-release and restored by a plain load.
- Restored from a caller's stack slot by xchg in a callee.

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 .../selftests/bpf/progs/verifier_live_stack.c      | 121 +++++++++++++++++++++
 1 file changed, 121 insertions(+)

diff --git a/tools/testing/selftests/bpf/progs/verifier_live_stack.c b/tools/testing/selftests/bpf/progs/verifier_live_stack.c
index 43f616ea90a9..ae4c5062cfa6 100644
--- a/tools/testing/selftests/bpf/progs/verifier_live_stack.c
+++ b/tools/testing/selftests/bpf/progs/verifier_live_stack.c
@@ -2696,6 +2696,127 @@ __naked void atomic_rmw(void)
 	: __clobber_all);
 }
 
+#ifdef CAN_USE_LOAD_ACQ_STORE_REL
+
+/*
+ * An 8-byte load-acquire from a spill slot restores the spilled register.
+ * The store through the restored pointer is a definite write to fp-8.
+ */
+SEC("socket")
+__log_level(2)
+__success
+__msg("r2 = load_acquire((u64 *)(r10 -16)){{.*}}use: fp0-16")
+__msg("*(u64 *)(r2 +0) = 42{{.*}}def: fp0-8 may_def: fp0-8")
+__naked void load_acquire_fills_spilled_ptr(void)
+{
+	asm volatile (
+	"r1 = r10;"
+	"r1 += -8;"
+	"*(u64 *)(r10 - 16) = r1;"		/* fp-16 = &fp-8 */
+	".8byte %[load_acquire_insn];"		/* r2 = load_acquire(fp-16) */
+	"*(u64 *)(r2 + 0) = 42;"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm_insn(load_acquire_insn,
+		     BPF_ATOMIC_OP(BPF_DW, BPF_LOAD_ACQ, BPF_REG_2, BPF_REG_10, -16))
+	: __clobber_all);
+}
+
+/* An 8-byte store-release spills the register, same as a plain store. */
+SEC("socket")
+__log_level(2)
+__success
+__msg("store_release((u64 *)(r10 -16), r1){{.*}}def: fp0-16")
+__msg("*(u64 *)(r2 +0) = 42{{.*}}def: fp0-8 may_def: fp0-8")
+__naked void store_release_spills_ptr(void)
+{
+	asm volatile (
+	"r1 = r10;"
+	"r1 += -8;"
+	".8byte %[store_release_insn];"		/* store_release(fp-16, r1): fp-16 = &fp-8 */
+	"r2 = *(u64 *)(r10 - 16);"
+	"*(u64 *)(r2 + 0) = 42;"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm_insn(store_release_insn,
+		     BPF_ATOMIC_OP(BPF_DW, BPF_STORE_REL, BPF_REG_10, BPF_REG_1, -16))
+	: __clobber_all);
+}
+
+#endif /* CAN_USE_LOAD_ACQ_STORE_REL */
+
+/* Stack fills through xchg, fetch_add and cmpxchg instructions. */
+SEC("socket")
+__log_level(2)
+__success
+__msg("r2 = atomic64_xchg((u64 *)(r10 -16), r2){{.*}}use: fp0-16{{.*}}may_def: fp0-16")
+__msg("*(u64 *)(r2 +0) = 42{{.*}}def: fp0-8 may_def: fp0-8")
+__msg("r3 = atomic64_fetch_add((u64 *)(r10 -16), r3){{.*}}use: fp0-16{{.*}}may_def: fp0-16")
+__msg("*(u64 *)(r3 +0) = 42{{.*}}def: fp0-8 may_def: fp0-8")
+__msg("r0 = atomic64_cmpxchg((u64 *)(r10 -16), r0, r4){{.*}}use: fp0-16{{.*}}may_def: fp0-16")
+__msg("*(u64 *)(r0 +0) = 42{{.*}}def: fp0-8 may_def: fp0-8")
+__naked void atomic_rmw_fills_spilled_ptr(void)
+{
+	asm volatile (
+	"r1 = r10;"
+	"r1 += -8;"
+	"*(u64 *)(r10 - 16) = r1;"		/* fp-16 = &fp-8 */
+	"r2 = 0;"
+	".8byte %[atomic_xchg];"		/* r2 = xchg(fp-16, r2) */
+	"*(u64 *)(r2 + 0) = 42;"
+	"*(u64 *)(r10 - 16) = r1;"
+	"r3 = 0;"
+	".8byte %[atomic_fetch_add];"		/* r3 = fetch_add(fp-16, r3) */
+	"*(u64 *)(r3 + 0) = 42;"
+	"*(u64 *)(r10 - 16) = r1;"
+	"r0 = 0;"
+	"r4 = 0;"
+	".8byte %[atomic_cmpxchg];"		/* r0 = cmpxchg(fp-16, r0, r4) */
+	"*(u64 *)(r0 + 0) = 42;"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm_insn(atomic_xchg, BPF_ATOMIC_OP(BPF_DW, BPF_XCHG, BPF_REG_10, BPF_REG_2, -16)),
+	  __imm_insn(atomic_fetch_add, BPF_ATOMIC_OP(BPF_DW, BPF_ADD | BPF_FETCH, BPF_REG_10, BPF_REG_3, -16)),
+	  __imm_insn(atomic_cmpxchg, BPF_ATOMIC_OP(BPF_DW, BPF_CMPXCHG, BPF_REG_10, BPF_REG_4, -16))
+	: __clobber_all);
+}
+
+/* xchg returning caller's &fp-8. */
+SEC("socket")
+__log_level(2)
+__success
+__msg("r2 = atomic64_xchg((u64 *)(r1 +0), r2){{.*}}use: fp0-16{{.*}}may_def: fp0-16")
+__msg("*(u64 *)(r2 +0) = 42{{.*}}; may_def: fp0-8")
+__naked void atomic_rmw_fills_parent_spilled_ptr(void)
+{
+	asm volatile (
+	"r1 = r10;"
+	"r1 += -8;"
+	"*(u64 *)(r10 - 16) = r1;"		/* fp-16 = &fp-8 */
+	"r1 = r10;"
+	"r1 += -16;"
+	"call atomic_rmw_fills_parent_spilled_ptr_callee;"
+	"r0 = 0;"
+	"exit;"
+	::: __clobber_all);
+}
+
+static __used __naked void atomic_rmw_fills_parent_spilled_ptr_callee(void)
+{
+	asm volatile (
+	"r2 = 0;"
+	".8byte %[atomic_xchg];"		/* r2 = xchg(r1, r2) == caller's &fp-8 */
+	"*(u64 *)(r2 + 0) = 42;"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm_insn(atomic_xchg, BPF_ATOMIC_OP(BPF_DW, BPF_XCHG, BPF_REG_1, BPF_REG_2, 0))
+	: __clobber_all);
+}
+
 SEC("socket")
 __success
 __naked void imprecise_fill_loses_cross_frame(void)

-- 
2.53.0

[PATCH bpf-next v3 43/51] selftests/bpf: tests for map special cases in bpf_helper_stack_access_bytes()

[Eduard Zingerman]

Tests for helper arguments whose access size or read/write effects
depend on the map:

- helper_map_output_known_size checks that for a known map access
  MEM_WRITE | MEM_UNINIT produces a must_write mark.
- helper_map_output_merged_sizes checks that when maps with different
  value sizes merge, MEM_WRITE | MEM_UNINIT produces use and may_def
  marks over the maximum value size, without a def mark.
- helper_bloom_peek_input checks that for bpf_map_peek_elem()
  on a bloom filter the buffer is treated as an input.

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 .../selftests/bpf/progs/verifier_live_stack.c      | 76 ++++++++++++++++++++++
 1 file changed, 76 insertions(+)

diff --git a/tools/testing/selftests/bpf/progs/verifier_live_stack.c b/tools/testing/selftests/bpf/progs/verifier_live_stack.c
index ae4c5062cfa6..0fbc43a5d4f4 100644
--- a/tools/testing/selftests/bpf/progs/verifier_live_stack.c
+++ b/tools/testing/selftests/bpf/progs/verifier_live_stack.c
@@ -3262,3 +3262,79 @@ __naked void helper_output_unknown_size(void)
 	:: __imm(bpf_get_prandom_u32), __imm(bpf_probe_read_kernel)
 	: __clobber_all);
 }
+
+struct {
+	__uint(type, BPF_MAP_TYPE_QUEUE);
+	__uint(max_entries, 1);
+	__type(value, __u64);
+} queue_8b SEC(".maps");
+
+struct {
+	__uint(type, BPF_MAP_TYPE_QUEUE);
+	__uint(max_entries, 1);
+	__type(value, __u64[2]);
+} queue_16b SEC(".maps");
+
+struct {
+	__uint(type, BPF_MAP_TYPE_BLOOM_FILTER);
+	__uint(max_entries, 16);
+	__type(value, __u64);
+} bloom_8b SEC(".maps");
+
+SEC("socket")
+__success __log_level(2)
+__msg("call bpf_map_pop_elem{{.*}}; def: fp0-16")
+__naked void helper_map_output_known_size(void)
+{
+	asm volatile (
+	"r1 = %[queue_8b] ll;"
+	"r2 = r10;"
+	"r2 += -16;"
+	"call %[bpf_map_pop_elem];"
+	"r0 = 0;"
+	"exit;"
+	:: __imm_addr(queue_8b), __imm(bpf_map_pop_elem)
+	: __clobber_all);
+}
+
+/* The maximum map value size is not a guaranteed output extent. */
+SEC("socket")
+__success __log_level(2)
+__msg("call bpf_map_pop_elem{{.*}}; use: fp0-16 fp0-24 may_def: fp0-16 fp0-24{{$}}")
+__naked void helper_map_output_merged_sizes(void)
+{
+	asm volatile (
+	"*(u64 *)(r10 - 16) = 0;"
+	"*(u64 *)(r10 - 24) = 0;"
+	"call %[bpf_get_prandom_u32];"
+	"r1 = %[queue_8b] ll;"
+	"if r0 == 0 goto 1f;"
+	"r1 = %[queue_16b] ll;"
+"1:"
+	"r2 = r10;"
+	"r2 += -24;"
+	"call %[bpf_map_pop_elem];"
+	"r0 = 0;"
+	"exit;"
+	:: __imm_addr(queue_8b), __imm_addr(queue_16b),
+	   __imm(bpf_get_prandom_u32), __imm(bpf_map_pop_elem)
+	: __clobber_all);
+}
+
+/* Bloom-filter peek treats buffer as an input despite the helper's output annotation. */
+SEC("socket")
+__success __log_level(2)
+__msg("call bpf_map_peek_elem{{.*}}; use: fp0-8{{$}}")
+__naked void helper_bloom_peek_input(void)
+{
+	asm volatile (
+	"*(u64 *)(r10 - 8) = 0;"
+	"r1 = %[bloom_8b] ll;"
+	"r2 = r10;"
+	"r2 += -8;"
+	"call %[bpf_map_peek_elem];"
+	"r0 = 0;"
+	"exit;"
+	:: __imm_addr(bloom_8b), __imm(bpf_map_peek_elem)
+	: __clobber_all);
+}

-- 
2.53.0

[PATCH bpf-next v3 44/51] selftests/bpf: tests for BTF-based global subprog stack liveness

[Eduard Zingerman]

Check that passing a stack pointer to a no-op global callee produces:
- use, def and may_def marks for a fixed-size memory argument.
- No stack effects for an untrusted BTF pointer argument.

This verifies that call effects follow the declared argument types
rather than the callee's body.

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 .../selftests/bpf/progs/verifier_live_stack.c      | 46 ++++++++++++++++++++++
 1 file changed, 46 insertions(+)

diff --git a/tools/testing/selftests/bpf/progs/verifier_live_stack.c b/tools/testing/selftests/bpf/progs/verifier_live_stack.c
index 0fbc43a5d4f4..3d88cb28b8fe 100644
--- a/tools/testing/selftests/bpf/progs/verifier_live_stack.c
+++ b/tools/testing/selftests/bpf/progs/verifier_live_stack.c
@@ -159,6 +159,52 @@ static __used __naked void write_first_param(void)
 	::: __clobber_all);
 }
 
+struct cnt8 { long v; };
+
+__noinline int global_noop_arg(struct cnt8 *p)
+{
+	return 0;
+}
+
+SEC("socket")
+__log_level(2)
+/* Call to a global function: the argument is described by its declared type. */
+__msg("stack use/def subprog#0 global_callee_arg (d0,cs0):")
+__msg("3: (85) call pc+{{.*}}; use: fp0-8 def: fp0-8 may_def: fp0-8")
+__naked void global_callee_arg(void)
+{
+	asm volatile (
+	"*(u64 *)(r10 - 8) = 0;"
+	"r1 = r10;"
+	"r1 += -8;"
+	"call global_noop_arg;"
+	"exit;"
+	::: __clobber_all);
+}
+
+struct task_struct { int pid; };
+
+__noinline int global_noop_btf_arg(struct task_struct *p __arg_untrusted)
+{
+	return 0;
+}
+
+SEC("socket")
+__success __log_level(2)
+/* Untrusted BTF pointer arguments need no caller stack state. */
+__msg("stack use/def subprog#0 global_callee_btf_arg (d0,cs0):")
+__msg("3: (85) call pc+{{[0-9]+}}\n")
+__naked void global_callee_btf_arg(void)
+{
+	asm volatile (
+	"*(u64 *)(r10 - 8) = 0;"
+	"r1 = r10;"
+	"r1 += -8;"
+	"call global_noop_btf_arg;"
+	"exit;"
+	::: __clobber_all);
+}
+
 /*
  * Cross-frame imprecise write: imprecise_frame_writer() receives a pointer
  * into the caller's frame (frame 0) but conditionally replaces it with a

-- 
2.53.0

Re: [PATCH bpf-next v3 44/51] selftests/bpf: tests for BTF-based global subprog stack liveness

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 2392 bytes --]

> selftests/bpf: tests for BTF-based global subprog stack liveness

TL;DR: Test issue: the "no stack effects" check in global_callee_btf_arg may
never fail, since the call-line __msg pattern also matches the main verifier
pass log, so a liveness regression could go undetected.

> diff --git a/tools/testing/selftests/bpf/progs/verifier_live_stack.c b/tools/testing/selftests/bpf/progs/verifier_live_stack.c
> --- a/tools/testing/selftests/bpf/progs/verifier_live_stack.c
> +++ b/tools/testing/selftests/bpf/progs/verifier_live_stack.c
> @@ -159,6 +159,52 @@ static __used __naked void write_first_param(void)

[ ... ]

> +struct task_struct { int pid; };
> +
> +__noinline int global_noop_btf_arg(struct task_struct *p __arg_untrusted)
> +{
> +	return 0;
> +}
> +
> +SEC("socket")
> +__success __log_level(2)
> +/* Untrusted BTF pointer arguments need no caller stack state. */
> +__msg("stack use/def subprog#0 global_callee_btf_arg (d0,cs0):")
> +__msg("3: (85) call pc+{{[0-9]+}}\n")
> +__naked void global_callee_btf_arg(void)

This isn't a bug, but can the absence check in global_callee_btf_arg ever
fail?

The pattern __msg("3: (85) call pc+{{[0-9]+}}\n") is matched against the
whole verifier log, not just the liveness dump.  If
bpf_global_subprog_stack_access_bytes() regressed and the call picked up
marks, print_instance() would emit:

  3: (85) call pc+1   ; use: fp0-8 may_def: fp0-8

which no longer matches.  But bpf_compute_live_registers() runs before
do_check_main(), and do_check() later logs the same instruction as:

    verbose(env, "%d: ", env->insn_idx);
    bpf_verbose_insn(env, insn);
    verbose(env, "\n");

That produces "3: (85) call pc+1\n", which satisfies the pattern regardless
of what the liveness dump printed.  The test is also __success, which does
not depend on the liveness marks.

The commit message says this test verifies "No stack effects for an untrusted
BTF pointer argument".  Would a __not_msg for the "call pc+" line with the
"; use:" / "may_def:" suffix, placed between the header __msg and a following
__msg, make the check effective?  That way the main-pass trace falls outside
the checked span.


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/37923187511

[PATCH bpf-next v3 45/51] selftests/bpf: test for register liveness of subprogram arguments

[Eduard Zingerman]

A static callee contributes the registers its code reads at the entry,
a global callee its declared arguments.

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 .../selftests/bpf/progs/compute_live_registers.c   | 29 ++++++++++++++++++++++
 1 file changed, 29 insertions(+)

diff --git a/tools/testing/selftests/bpf/progs/compute_live_registers.c b/tools/testing/selftests/bpf/progs/compute_live_registers.c
index b274a20ed7bd..b0aaf5961ac9 100644
--- a/tools/testing/selftests/bpf/progs/compute_live_registers.c
+++ b/tools/testing/selftests/bpf/progs/compute_live_registers.c
@@ -414,6 +414,35 @@ static __used __naked int aux1(void)
 		::: __clobber_all);
 }
 
+static __noinline __used int static_two(int a, int b)
+{
+	return a;
+}
+
+__noinline __used int global_two(int a, int b)
+{
+	return a;
+}
+
+SEC("socket")
+__log_level(2)
+/* a static callee reads r1 only, a global callee its declared arguments */
+__msg("2: .1........ (85) call pc+")
+__msg("5: .12....... (85) call pc+")
+__naked void subprog_declared_args(void)
+{
+	asm volatile (
+		"r1 = 1;"
+		"r2 = 2;"
+		"call static_two;"
+		"r1 = 1;"
+		"r2 = 2;"
+		"call global_two;"
+		"r0 = 0;"
+		"exit;"
+		::: __clobber_all);
+}
+
 SEC("socket")
 __log_level(2)
 __msg("0: .......... (b7) r1 = 1")

-- 
2.53.0

[PATCH bpf-next v3 46/51] selftests/bpf: tests for register base/step arithmetic

[Eduard Zingerman]

- step_arith{32,64}_no_overflow: non-overflowing +, *, <<
  operations maintain step and base.
- step_neg_value_range: Preserve the signed range of {-3, 0, 3, 6}
  when intersecting with a non-power-of-two step across zero.
- step_arith{32,64}_overflow: Reset the stride and preserve valid
  bounds when a {32,64}-bit operation wraps.
- step_mov32_truncate: Reset the stride when MOV32 discards nonzero
  upper bits from positive values.
- step_mov32_negative: Reset the stride when MOV32 zero-extends
  the low 32 bits of negative signed values.
- step_movsx: Reset the stride when sign extension changes the
  register value.
- step_movsx_bounds_on_line: Keep bounds and stride when sign
  extension does not change the register value.
- step_bounds_resync_after_arith: Snap bounds refined from var_off
  back onto the line.
- step_linked_regs_add64: Translate both bounds and base when
  propagating a comparison's refinement through a scalar-ID delta.

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 tools/testing/selftests/bpf/prog_tests/verifier.c  |   2 +
 .../selftests/bpf/progs/verifier_bounds_step.c     | 363 +++++++++++++++++++++
 2 files changed, 365 insertions(+)

diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c
index 460ad10ddc02..4f3af1cb998b 100644
--- a/tools/testing/selftests/bpf/prog_tests/verifier.c
+++ b/tools/testing/selftests/bpf/prog_tests/verifier.c
@@ -20,6 +20,7 @@
 #include "verifier_basic_stack.skel.h"
 #include "verifier_bitfield_write.skel.h"
 #include "verifier_bounds.skel.h"
+#include "verifier_bounds_step.skel.h"
 #include "verifier_bounds_deduction.skel.h"
 #include "verifier_bounds_deduction_non_const.skel.h"
 #include "verifier_bounds_mix_sign_unsign.skel.h"
@@ -205,6 +206,7 @@ void test_verifier_arena_globals2(void)       { RUN(verifier_arena_globals2); }
 void test_verifier_basic_stack(void)          { RUN(verifier_basic_stack); }
 void test_verifier_bitfield_write(void)       { RUN(verifier_bitfield_write); }
 void test_verifier_bounds(void)               { RUN(verifier_bounds); }
+void test_verifier_bounds_step(void)          { RUN(verifier_bounds_step); }
 void test_verifier_bounds_deduction(void)     { RUN(verifier_bounds_deduction); }
 void test_verifier_bounds_deduction_non_const(void)     { RUN(verifier_bounds_deduction_non_const); }
 void test_verifier_bounds_mix_sign_unsign(void) { RUN(verifier_bounds_mix_sign_unsign); }
diff --git a/tools/testing/selftests/bpf/progs/verifier_bounds_step.c b/tools/testing/selftests/bpf/progs/verifier_bounds_step.c
new file mode 100644
index 000000000000..3a0d22aa4fb8
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/verifier_bounds_step.c
@@ -0,0 +1,363 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+/* Tests for the linear "base + step * k" description tracked per scalar register. */
+
+#define __no_step __not_msg("step=") __msg("\n")
+
+/*
+ * Safe ALU64 arithmetic preserves a non-power-of-two stride and scales its base:
+ * step=0+3 -> step=1+3 -> step=2+6 -> step=4+12.
+ */
+SEC("socket")
+__success __log_level(2)
+__msg("r0 *= 3 {{.*}}step=0+3)")
+__msg("r0 += 1 {{.*}}step=1+3)")
+__msg("r0 *= 2 {{.*}}step=2+6)")
+__msg("r0 <<= 1 {{.*}}step=4+12)")
+__naked void step_arith64_no_overflow(void)
+{
+	asm volatile ("					\
+	call %[bpf_get_prandom_u32];			\
+	r0 &= 0xff;					\
+	r0 *= 3;					\
+	r0 += 1;					\
+	r0 *= 2;					\
+	r0 <<= 1;					\
+	r0 = 0;						\
+	exit;						\
+"	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+/*
+ * Safe ALU32 multiplication and addition preserve the stride and scale its base:
+ * step=0+3 -> step=1+3 -> step=2+6. MOV32 preserves the result.
+ * LSH32 currently resets the stride, but must preserve the correct upper bound.
+ */
+SEC("socket")
+__success __log_level(2)
+__msg("w0 *= 3 {{.*}}step=0+3)")
+__msg("w0 += 1 {{.*}}step=1+3)")
+__msg("w0 *= 2 {{.*}}step=2+6)")
+__msg("w1 = w0 {{.*}}R0=scalar({{.*}}step=2+6) R1=scalar({{.*}}step=2+6)")
+__msg("w0 <<= 1 {{.*}}smax=umax=smax32=umax32=3064,") __no_step
+__naked void step_arith32_no_overflow(void)
+{
+	asm volatile ("					\
+	call %[bpf_get_prandom_u32];			\
+	r0 &= 0xff;					\
+	w0 *= 3;					\
+	w0 += 1;					\
+	w0 *= 2;					\
+	w1 = w0;					\
+	w0 <<= 1;					\
+	r0 = 0;						\
+	exit;						\
+"	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+/*
+ * Sign-crossing range with a non-power-of-2 step. After "*= 3; += -3" the value
+ * set is {-3, 0, 3, 6}. The line description is tracked in signed space, so the
+ * intersection keeps smax=6.
+ */
+SEC("socket")
+__success __log_level(2)
+__msg("r0 += -3 {{.*}}smax=smax32=6,step=0+3)")
+__naked void step_neg_value_range(void)
+{
+	asm volatile ("					\
+	call %[bpf_get_prandom_u32];			\
+	r0 &= 3;					\
+	r0 *= 3;					\
+	r0 += -3;					\
+	r0 = 0;						\
+	exit;						\
+"	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+/*
+ * 0x49249249 * 7 wraps to U32_MAX; inferring step=0+7 would incorrectly
+ * cap the result at 0xfffffffc instead of U32_MAX.
+ * Adding 4 to {0xfffffffc, U32_MAX} wraps to {0, 3}; keeping step=1+3
+ * incorrectly narrows the result to 1.
+ * Shifting the same input left by 1 gives {0xfffffff8, 0xfffffffe};
+ * inferring step=0+6 incorrectly narrows the result to 0xfffffffc.
+ */
+SEC("socket")
+__success __log_level(2)
+__msg("w0 *= 7 {{.*}}umax=0xffffffff,") __no_step
+__msg("r2 = r0 {{.*}}R0=scalar({{.*}}step=0+3) R2=scalar({{.*}}step=0+3)")
+__msg("w0 += 4 {{.*}}smax=umax=smax32=umax32=3,") __no_step
+__msg("w2 <<= 1 {{.*}}smax=umax=umax32=0xfffffffe,") __no_step
+__naked void step_arith32_overflow(void)
+{
+	asm volatile ("					\
+	call %[bpf_get_prandom_u32];			\
+	w0 *= 7;					\
+	call %[bpf_get_prandom_u32];			\
+	r0 &= 1;					\
+	r0 *= 3;					\
+	r1 = 0xfffffffc ll;				\
+	r0 += r1;					\
+	r2 = r0;					\
+	w0 += 4;					\
+	w2 <<= 1;					\
+	r0 = 0;						\
+	exit;						\
+"	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+/*
+ * Multiplying {0x2aaaaaaaaaaaaaab, 0x2aaaaaaaaaaaaaac} by 3 crosses S64_MAX
+ * and gives {S64_MIN + 1, S64_MIN + 4}. Keeping step=0+3 excludes both.
+ *
+ * Adding 5 to {S64_MAX - 4, S64_MAX - 1} gives {S64_MIN, S64_MIN + 3}.
+ * Updating the line to step=2+3 incorrectly narrows it to S64_MIN + 1.
+ *
+ * Adding -5 to {S64_MIN + 1, S64_MIN + 4} wraps to
+ * {S64_MAX - 3, S64_MAX}. The stale base 0 modulo 3 excludes S64_MAX.
+ *
+ * Shifting {0x4000000000000008, 0x400000000000000b} left by 1 gives
+ * {S64_MIN + 16, S64_MIN + 22}. Keeping step=0+6 excludes both.
+ */
+SEC("socket")
+__success __log_level(2)
+__msg("r0 *= 3 {{.*}}smax=0x8000000000000004,") __no_step
+__msg("r0 += r1 {{.*}}step=0+3)")
+__msg("r0 += 5 {{.*}}smax=0x8000000000000003,") __no_step
+__msg("r0 += r1 {{.*}}step=2+3)")
+__msg("r0 += -5 {{.*}}umax=0x7fffffffffffffff,") __no_step
+__msg("r0 += r1 {{.*}}step=0+3)")
+__msg("r0 <<= 1 {{.*}}smax=0x8000000000000016,") __no_step
+__naked void step_arith64_overflow(void)
+{
+	asm volatile ("					\
+	call %[bpf_get_prandom_u32];			\
+	r0 &= 1;					\
+	r2 = r0;					\
+	r1 = 0x2aaaaaaaaaaaaaab ll;			\
+	r0 += r1;					\
+	r0 *= 3;					\
+	r2 *= 3;					\
+	r0 = r2;					\
+	r1 = 0x7ffffffffffffffb ll;			\
+	r0 += r1;					\
+	r0 += 5;					\
+	r0 = r2;					\
+	r1 = 0x8000000000000001 ll;			\
+	r0 += r1;					\
+	r0 += -5;					\
+	r0 = r2;					\
+	r1 = 0x4000000000000008 ll;			\
+	r0 += r1;					\
+	r0 <<= 1;					\
+	r0 = 0;						\
+	exit;						\
+"	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+/*
+ * MOV32 maps {2^32, 2^32 + 3} to {0, 3}. The old base 1 modulo 3
+ * excludes both results.
+ */
+SEC("socket")
+__success __log_level(2)
+__msg("r0 += r1 {{.*}}step=1+3)")
+__msg("w0 = w0 {{.*}}smax=umax=smax32=umax32=3,") __no_step
+__naked void step_mov32_truncate(void)
+{
+	asm volatile ("					\
+	call %[bpf_get_prandom_u32];			\
+	r0 &= 1;					\
+	r0 *= 3;					\
+	r1 = 0x100000000 ll;				\
+	r0 += r1;					\
+	w0 = w0;					\
+	r0 = 0;						\
+	exit;						\
+"	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+/*
+ * MOV32 maps {-4, -1} to {0xfffffffc, U32_MAX}. The old base 2
+ * modulo 3 excludes U32_MAX.
+ */
+SEC("socket")
+__success __log_level(2)
+__msg("r0 += -4 {{.*}}step=2+3)")
+__msg("w0 = w0 {{.*}}umax=0xffffffff,") __no_step
+__naked void step_mov32_negative(void)
+{
+	asm volatile ("					\
+	call %[bpf_get_prandom_u32];			\
+	r0 &= 1;					\
+	r0 *= 3;					\
+	r0 += -4;					\
+	w0 = w0;					\
+	r0 = 0;						\
+	exit;						\
+"	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+/*
+ * Sign-extending {128, 131} from s8 gives {-128, -125}.
+ * Keeping base 2 modulo 3 incorrectly narrows the result to -127.
+ *
+ * Sign-extending {32768, 32771} from s16 gives {-32768, -32765}.
+ * Keeping base 2 modulo 3 incorrectly narrows the result to -32767.
+ *
+ * Sign-extending {2^31, 2^31 + 3} from s32 gives {S32_MIN, S32_MIN + 3}.
+ * Keeping base 2 modulo 3 incorrectly narrows the result to S32_MIN + 1.
+ */
+SEC("socket")
+__success __log_level(2)
+__msg("r0 += 128 {{.*}}step=2+3)")
+__msg("r0 = (s8)r0 {{.*}}smax=smax32=-125,") __no_step
+__msg("r0 += 32768 {{.*}}step=2+3)")
+__msg("r0 = (s16)r0 {{.*}}smax=smax32=-32765,") __no_step
+__msg("r0 += r1 {{.*}}step=2+3)")
+__msg("r0 = (s32)r0 {{.*}}smax=0xffffffff80000003,") __no_step
+__naked void step_movsx(void)
+{
+	asm volatile ("					\
+	call %[bpf_get_prandom_u32];			\
+	r0 &= 1;					\
+	r0 *= 3;					\
+	r2 = r0;					\
+	r0 += 128;					\
+	r0 = (s8)r0;					\
+	r0 = r2;					\
+	r0 += 32768;					\
+	r0 = (s16)r0;					\
+	r0 = r2;					\
+	r1 = 0x80000000 ll;				\
+	r0 += r1;					\
+	r0 = (s32)r0;					\
+	r0 = 0;						\
+	exit;						\
+"	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+/*
+ * R1 = R0 + 1 shares R0's scalar ID. Refining R0 to {0, 3, 6} must
+ * translate its line as well as its bounds when updating R1 to {1, 4, 7}.
+ * Copying R0's base 0 modulo 3 excludes the reachable value 7.
+ */
+SEC("socket")
+__success __log_level(2)
+__msg("r1 += 1 {{.*}}step=1+3)")
+__msg("if r0 > 0x6 {{.*}}R1=scalar({{.*}}smin=umin=smin32=umin32=1,smax=umax=smax32=umax32=7,{{.*}}step=1+3)")
+__naked void step_linked_regs_add64(void)
+{
+	asm volatile ("					\
+	call %[bpf_get_prandom_u32];			\
+	r0 &= 3;					\
+	r0 *= 3;					\
+	r1 = r0;					\
+	r1 += 1;					\
+	if r0 > 6 goto l_out_%=;			\
+	r0 = r1;					\
+l_out_%=:						\
+	r0 = 0;						\
+	exit;						\
+"	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+/*
+ * r0 &= 7       -> [0, 7]
+ * r0 *= 24      -> [0, 168]   step=0+24
+ * r0 += 8       -> [8, 176]   step=8+24
+ * r0 & 0x10     -> [32, 176]  step=8+24  (var_off alone: umin=16)
+ * r0 > 80       -> [104, 176] step=8+24  (var_off alone: umin=88)
+ * r1 += 24      -> [128, 200] step=8+24
+ * Bounds and step stay in sync after each refinement.
+ */
+SEC("socket")
+__success __log_level(2)
+__msg("r0 += 8 {{.*}}smin=umin=smin32=umin32=8,smax=umax=smax32=umax32=176,{{.*}}step=8+24)")
+__msg("from 4 to 6: R0=scalar(smin=umin=smin32=umin32=32,smax=umax=smax32=umax32=176,{{.*}}step=8+24)")
+__msg("r1 = r0 {{.*}}R1=scalar(id={{[0-9]+}},smin=umin=smin32=umin32=104,smax=umax=smax32=umax32=176,{{.*}}step=8+24)")
+__msg("r1 += 24 {{.*}}R1=scalar({{.*}}smin=umin=smin32=umin32=128,smax=umax=smax32=umax32=200,{{.*}}step=8+24)")
+__naked void step_bounds_resync_after_arith(void)
+{
+	asm volatile ("					\
+	call %[bpf_get_prandom_u32];			\
+	r0 &= 7;					\
+	r0 *= 24;					\
+	r0 += 8;					\
+	if r0 & 0x10 goto l_set_%=;			\
+	goto l_out_%=;					\
+l_set_%=:						\
+	if r0 <= 80 goto l_out_%=;			\
+	r1 = r0;					\
+	r1 += 24;					\
+	*(u64 *)(r10 - 8) = r1;				\
+l_out_%=:						\
+	r0 = 0;						\
+	exit;						\
+"	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+/*
+ * r0 &= 7       -> [0, 7]
+ * r0 *= 24      -> [0, 168]    step=0+24
+ * r1 = (s16)r0  -> [0, 168]    step=0+24
+ * r0 += -24     -> [-24, 144]  step=0+24
+ * r2 = (s16)r0  -> [-24, 144]  step=0+24
+ * r3 = (s8)r0   -> [-128, 127] no step
+ * w4 = (s16)w1  -> [0, 168]    step=0+24
+ * w5 = (s8)w1   -> [-128, 127] no step
+ * Bounds and step stay in sync, the step is dropped when the value does not fit.
+ */
+SEC("socket")
+__success __log_level(2)
+__msg("r1 = (s16)r0 {{.*}}R1=scalar(id={{[0-9]+}},smin=smin32=0,smax=umax=smax32=umax32=168,{{.*}}step=0+24)")
+__msg("r2 = (s16)r0 {{.*}}R2=scalar(smin=smin32=-24,smax=smax32=144,{{.*}}step=0+24)")
+__msg("r3 = (s8)r0 {{.*}}R3=scalar(smin=smin32=-128,smax=smax32=127)") __no_step
+__msg("w4 = (s16)w1 {{.*}}R4=scalar(id={{[0-9]+}},smin=smin32=0,smax=umax=smax32=umax32=168,{{.*}}step=0+24)")
+__msg("w5 = (s8)w1 {{.*}}R5=scalar({{.*}}smin32=-128,smax32=127,{{.*}})") __no_step
+__naked void step_movsx_bounds_on_line(void)
+{
+	asm volatile ("					\
+	call %[bpf_get_prandom_u32];			\
+	r0 &= 7;					\
+	r0 *= 24;					\
+	r1 = (s16)r0;					\
+	r0 += -24;					\
+	r2 = (s16)r0;					\
+	r3 = (s8)r0;					\
+	w4 = (s16)w1;					\
+	w5 = (s8)w1;					\
+	r0 = 0;						\
+	exit;						\
+"	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+char _license[] SEC("license") = "GPL";

-- 
2.53.0

Re: [PATCH bpf-next v3 46/51] selftests/bpf: tests for register base/step arithmetic

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 3272 bytes --]

> selftests/bpf: tests for register base/step arithmetic

TL;DR: In verifier_bounds_step.c, the __no_step check on the "w5 = (s8)w1"
pattern looks unable to fail, since the trailing {{.*}}) swallows any "step="
text, so a missing step reset in sign-extension would go undetected.

> diff --git a/tools/testing/selftests/bpf/progs/verifier_bounds_step.c b/tools/testing/selftests/bpf/progs/verifier_bounds_step.c
> new file mode 100644
> index 000000000000..3a0d22aa4fb8
> --- /dev/null
> +++ b/tools/testing/selftests/bpf/progs/verifier_bounds_step.c
> @@ -0,0 +1,363 @@

[ ... ]

> + * w4 = (s16)w1  -> [0, 168]    step=0+24
> + * w5 = (s8)w1   -> [-128, 127] no step
> + * Bounds and step stay in sync, the step is dropped when the value does not fit.
> + */
> +SEC("socket")
> +__success __log_level(2)

[ ... ]

> +__msg("r3 = (s8)r0 {{.*}}R3=scalar(smin=smin32=-128,smax=smax32=127)") __no_step
> +__msg("w4 = (s16)w1 {{.*}}R4=scalar(id={{[0-9]+}},smin=smin32=0,smax=umax=smax32=umax32=168,{{.*}}step=0+24)")
> +__msg("w5 = (s8)w1 {{.*}}R5=scalar({{.*}}smin32=-128,smax32=127,{{.*}})") __no_step

Can the __no_step check after the w5 = (s8)w1 pattern ever fail?  It looks
like it would pass even if a step were left on R5.

__no_step expands to __not_msg("step=") __msg("\n"), and
match_negative_msgs() in test_loader.c only looks for "step=" between the
end of the previous positive match and the next newline.  The positive
pattern here ends in {{.*}}), and with leftmost-longest matching that
trailing .*\) runs to the last ')' on the line.

R5 is the highest-numbered scratched register at this insn, so the last
')' is the one closing R5=scalar(...).  print_reg_state() emits
"step=%d+%d" just before that closing ')', so any step text is swallowed
by the {{.*}} and the span left for the negative check is empty.

For example, if the reg_step_reset() in coerce_subreg_to_size_sx() were
removed, R5 would keep step=0+24.  zext_32_to_64() would not drop it
either, because reg_umax() at that point is still the copied 64-bit umax
of 168.  The log line would then be:

  R5=scalar(smin=0,smax=umax=0xfffffff0,smin32=-128,smax32=127,var_off=(0x0; 0xffffffff),step=0+24)

and the test would still pass.  I checked this with libc regcomp/regexec
on a log line built the same way.

The comment above the test says "w5 = (s8)w1 -> [-128, 127] no step", but
no other test in the file covers the coerce_subreg_to_size_sx() reset:
step_movsx only uses 64-bit sign extension, and w4 = (s16)w1 takes the
no-op path.

The R3 check a couple of lines earlier works only because its pattern ends
in an explicit "127)".  Would dropping the trailing {{.*}}) from the R5
pattern, e.g.:

  __msg("w5 = (s8)w1 {{.*}}R5=scalar({{.*}}smin32=-128,smax32=127,") __no_step

let __no_step see the rest of R5's state (var_off=...) and fail when a
step is present?  I checked this as well.  The later series commit
c7b36680c7d9 ("selftests/bpf: tests for register base/step state
pruning") only appends to this file and does not change this pattern.


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/37923187511

[PATCH bpf-next v3 47/51] selftests/bpf: tests for register base/step state pruning

[Eduard Zingerman]

Tests for the base/step reasoning in states.c:range_within():
- step_prune_hit_multiple: prune a step-4 range contained in a cached
  step-2 range when the scalar is used to address the stack.
- step_prune_miss_non_multiple: do not let a cached step-2 range hide a
  step-3 path reaching division by zero.
- step_prune_hit_const_on_line: prune constant 6 against a cached
  base-0, step-3 range.
- step_prune_miss_const_off_line: do not prune constant 7 against that
  range even though its bounds and tnum admit the value.

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 .../selftests/bpf/progs/verifier_bounds_step.c     | 181 +++++++++++++++++++++
 1 file changed, 181 insertions(+)

diff --git a/tools/testing/selftests/bpf/progs/verifier_bounds_step.c b/tools/testing/selftests/bpf/progs/verifier_bounds_step.c
index 3a0d22aa4fb8..c9ee1c7efcbb 100644
--- a/tools/testing/selftests/bpf/progs/verifier_bounds_step.c
+++ b/tools/testing/selftests/bpf/progs/verifier_bounds_step.c
@@ -360,4 +360,185 @@ __naked void step_movsx_bounds_on_line(void)
 	: __clobber_all);
 }
 
+struct step_val {
+	__u8 data[1024];
+};
+
+struct {
+	__uint(type, BPF_MAP_TYPE_ARRAY);
+	__uint(max_entries, 1);
+	__type(key, __u32);
+	__type(value, struct step_val);
+} step_map SEC(".maps");
+
+/* Old register [4..130, step 2] should prune cur register [8..68, step 4]. */
+SEC("socket")
+__success __log_level(2)
+__msg("7: (27) r6 *= 4                       ; R6=scalar({{.*}}umin32=8,{{.*}}umax32=68,{{.*}},step=0+4)")
+__msg("10: (27) r7 *= 2                      ; R7=scalar({{.*}}umin32=4,{{.*}}umax32=130,{{.*}},step=0+2)")
+__msg("11: (25) if r0 > 0x2a goto pc+1")
+__msg("from 11 to 13: safe")
+__flag(BPF_F_TEST_STATE_FREQ)
+__naked void step_prune_hit_multiple(void)
+{
+	asm volatile ("					\
+	call %[bpf_get_prandom_u32];			\
+	r6 = r0;					\
+	call %[bpf_get_prandom_u32];			\
+	r7 = r0;					\
+	call %[bpf_get_prandom_u32];			\
+	r6 &= 0x0f;					\
+	r6 += 2;					\
+	r6 *= 4;					\
+	r7 &= 0x3f;					\
+	r7 += 2;					\
+	r7 *= 2;					\
+	if r0 > 42 goto 1f;	/* can't predict */	\
+	r6 = r7;		/* step=2 explored first, step=4 explored next */ \
+1:	r0 = r10;					\
+	r6 = -r6;					\
+	r0 += r6;					\
+	*(u8 *)(r0 + 0) = 7;	/* force r6 precise */	\
+	exit;						\
+"	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+/*
+ * Old register [0..189, step 3] should not prune cur register [0..30, step 2].
+ * Bounds and tnum containment allow pruning; step divisibility must prevent it.
+ */
+SEC("socket")
+__success __log_level(2)
+__msg("6: (27) r6 *= 2                       ; R6=scalar({{.*}}smin32=0,{{.*}}umax32=30,{{.*}},step=0+2)")
+__msg("8: (27) r7 *= 3                       ; R7=scalar({{.*}}smin32=0,{{.*}}umax32=189,{{.*}},step=0+3)")
+__msg("9: (25) if r0 > 0x2a goto pc+1")
+__msg("17: (95) exit")
+__msg("from 9 to 11: R6=scalar({{.*}},step=0+2)")
+__flag(BPF_F_TEST_STATE_FREQ)
+__naked void step_prune_miss_non_multiple(void)
+{
+	asm volatile ("					\
+	call %[bpf_get_prandom_u32];			\
+	r6 = r0;					\
+	call %[bpf_get_prandom_u32];			\
+	r7 = r0;					\
+	call %[bpf_get_prandom_u32];			\
+	r6 &= 0x0f;					\
+	r6 *= 2;					\
+	r7 &= 0x3f;					\
+	r7 *= 3;					\
+	if r0 > 42 goto 1f;	/* can't predict */	\
+	r6 = r7;		/* step=3 explored first, step=2 explored next */ \
+1:							\
+	r2 = r10;					\
+	r6 += 1;					\
+	r6 = -r6;					\
+	r2 += r6;					\
+	*(u8 *)(r2 + 0) = 7;	/* force r6 precise */	\
+	r0 = 0;						\
+	exit;						\
+"	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+/*
+ * Constant current register lying on the cached line: cached is {0,3,6,...}
+ * (step 3, base 0), current is the constant 6. range_within() takes the
+ * constant branch: imod(6, 3) == base 0, so cur is on the line and the
+ * (precise) state is pruned -> "safe".
+ */
+SEC("socket")
+__success __log_level(2)
+__flag(BPF_F_TEST_STATE_FREQ)
+__msg("14: (27) r1 *= 3")		/* cached path: step 3 line */
+__msg("16: (b7) r1 = 6")		/* current path: const 6, on the line */
+__msg("17: safe")			/* pruned at join: imod(6, 3) == 0 */
+__naked void step_prune_hit_const_on_line(void)
+{
+	asm volatile ("					\
+	call %[bpf_get_prandom_u32];			\
+	r6 = r0;					\
+	r1 = 0;						\
+	*(u32*)(r10 - 4) = r1;				\
+	r2 = r10;					\
+	r2 += -4;					\
+	r1 = %[step_map] ll;				\
+	call %[bpf_map_lookup_elem];			\
+	if r0 == 0 goto l_out_%=;			\
+	r7 = r0;					\
+	r1 = r6;					\
+	r1 &= 0xff;					\
+	if r6 > 0 goto l_cur_%=;			\
+	r1 *= 3;			/* old: step 3 */	\
+	goto l_join_%=;					\
+l_cur_%=:						\
+	r1 = 6;				/* cur: const on line */	\
+l_join_%=:						\
+	r0 = r7;					\
+	r0 += r1;			/* r1 forced precise */	\
+	r2 = *(u8 *)(r0 + 0);				\
+l_out_%=:						\
+	r0 = 0;						\
+	exit;						\
+"	:
+	: __imm(bpf_get_prandom_u32),
+	  __imm(bpf_map_lookup_elem),
+	  __imm_addr(step_map)
+	: __clobber_all);
+}
+
+/*
+ * Constant current register NOT on the cached line: cached is {0,3,6,...}
+ * (step 3, base 0), current is the constant 7. imod(7, 3) == 1 != base 0,
+ * so range_within() fails and the join is traversed again. A power-of-two
+ * step is avoided on purpose: with step 3 the tnum is loose enough to admit
+ * 7, so imod() is
+ * the sole check that rejects it. No failure shape is possible here: 7 is
+ * within the line's bounds and tnum, and the cached line path already
+ * verifies the whole outro, so an eager prune could not miss an error.
+ */
+SEC("socket")
+__success __log_level(2)
+__flag(BPF_F_TEST_STATE_FREQ)
+__msg("14: (27) r1 *= 3")		/* cached path: step 3 line */
+__msg("16: (b7) r1 = 7")		/* current path: const 7, off the line */
+/* not pruned: current continues past the join with the constant offset 7 */
+__msg("19: R0=map_value({{.*}}imm=7) R1=7")
+__naked void step_prune_miss_const_off_line(void)
+{
+	asm volatile ("					\
+	call %[bpf_get_prandom_u32];			\
+	r6 = r0;					\
+	r1 = 0;						\
+	*(u32*)(r10 - 4) = r1;				\
+	r2 = r10;					\
+	r2 += -4;					\
+	r1 = %[step_map] ll;				\
+	call %[bpf_map_lookup_elem];			\
+	if r0 == 0 goto l_out_%=;			\
+	r7 = r0;					\
+	r1 = r6;					\
+	r1 &= 0xff;					\
+	if r6 > 0 goto l_cur_%=;			\
+	r1 *= 3;			/* old: step 3 */	\
+	goto l_join_%=;					\
+l_cur_%=:						\
+	r1 = 7;				/* cur: const off line */	\
+l_join_%=:						\
+	r0 = r7;					\
+	r0 += r1;			/* r1 forced precise */	\
+	r2 = *(u8 *)(r0 + 0);				\
+l_out_%=:						\
+	r0 = 0;						\
+	exit;						\
+"	:
+	: __imm(bpf_get_prandom_u32),
+	  __imm(bpf_map_lookup_elem),
+	  __imm_addr(step_map)
+	: __clobber_all);
+}
+
 char _license[] SEC("license") = "GPL";

-- 
2.53.0

Re: [PATCH bpf-next v3 47/51] selftests/bpf: tests for register base/step state pruning

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] bpf: selftest step_prune_hit_multiple fails verification due to R0 leaking stack pointer
--

commit f9da8ee53b52465f66b36e329c27712344008e2e
Author: Eduard Zingerman <eddyz87@gmail.com>

selftests/bpf: tests for register base/step state pruning

This commit adds tests for base/step reasoning in range_within(), including
tests like step_prune_hit_multiple and step_prune_miss_non_multiple.

> diff --git a/tools/testing/selftests/bpf/progs/verifier_bounds_step.c b/tools/testing/selftests/bpf/progs/verifier_bounds_step.c
> index 3a0d22aa4fb89..c9ee1c7efcbb8 100644
> --- a/tools/testing/selftests/bpf/progs/verifier_bounds_step.c
> +++ b/tools/testing/selftests/bpf/progs/verifier_bounds_step.c

[ ... ]

> +__naked void step_prune_hit_multiple(void)
> +{
> +	asm volatile ("					\
> +	call %[bpf_get_prandom_u32];			\
> +	r6 = r0;					\
> +	call %[bpf_get_prandom_u32];			\
> +	r7 = r0;					\
> +	call %[bpf_get_prandom_u32];			\
> +	r6 &= 0x0f;					\
> +	r6 += 2;					\
> +	r6 *= 4;					\
> +	r7 &= 0x3f;					\
> +	r7 += 2;					\
> +	r7 *= 2;					\
> +	if r0 > 42 goto 1f;	/* can't predict */	\
> +	r6 = r7;		/* step=2 explored first, step=4 explored next */ \
> +1:	r0 = r10;					\
> +	r6 = -r6;					\
> +	r0 += r6;					\
> +	*(u8 *)(r0 + 0) = 7;	/* force r6 precise */	\
> +	exit;						\

[Severity: Medium]
Will this selftest fail verification due to leaking a stack pointer?

It appears that r0 is assigned a stack address (r0 = r10; r0 += r6;), but is
not reset to a valid scalar value before the exit instruction. As a
SEC("socket") program, the verifier requires r0 to be a valid scalar upon
exit, and returning a pointer to the stack will cause the verifier to reject
the program with an 'R0 leaks addr' error.

Since this test is marked with __success, wouldn't this verifier rejection
cause the test case to fail? Should r0 be reset (e.g., r0 = 0;) before exit,
similar to how it is done in step_prune_miss_non_multiple()?

> +"	:
> +	: __imm(bpf_get_prandom_u32)
> +	: __clobber_all);
> +}

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261009-scev-minimal-rebase-v3-0-bcba9bc53ac9@gmail.com?part=47

[PATCH bpf-next v3 48/51] selftests/bpf: tests for varying offset access to PTR_TO_BTF_ID

[Eduard Zingerman]

Verifier tests for accessing arrays at a varying offset through a
PTR_TO_BTF_ID pointer, e.g. a[i].b where 'i' is a register with a
varying value:
- bpf_obj_new() is used to obtain a PTR_TO_BTF_ID to a program-defined
  struct
- bpf_get_current_task_btf() is used to obtain a PTR_TO_BTF_ID from
  the kernel BTF.

The cases cover:
- in-bounds access to a field of an array of structs;
- out-of-bounds access where the maximal offset runs past the array;
- a scalar (long) array with a matching stride;
- a byte array (step 1);
- a 4-byte read whose maximal offset spills past the array end;
- an unaligned step that is not a whole number of elements;
- both dimensions of a 2D array;
- an array of pointers within a program-allocated object, where reading a
  pointer member yields a SCALAR_VALUE but the varying offset is still
  validated against the array bounds (in-bounds and out-of-bounds);
- a varying access into an array embedded in a kernel BTF type
  (task_struct's comm[]), covering the non-allocated, trusted-pointer path
  (in-bounds and out-of-bounds);
- an array nested below a struct member at a non-zero offset, exercising
  the offset telescoping done while walking across a struct boundary
  (in-bounds and out-of-bounds);
- a trailing flexible array member, whose unbounded tail makes a varying
  access have no upper bound to exceed. A program-defined struct cannot be
  used here because bpf_obj_new() forbids flexible array members, so a
  kernel type (struct vring_used, with its 'ring[]' tail) is used
  (in-bounds, and an unaligned step that is not a whole number of
   elements).

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 tools/testing/selftests/bpf/prog_tests/verifier.c  |   2 +
 .../bpf/progs/verifier_btf_array_access.c          | 576 +++++++++++++++++++++
 2 files changed, 578 insertions(+)

diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c
index 4f3af1cb998b..2f05a0ce829f 100644
--- a/tools/testing/selftests/bpf/prog_tests/verifier.c
+++ b/tools/testing/selftests/bpf/prog_tests/verifier.c
@@ -27,6 +27,7 @@
 #include "verifier_bpf_get_stack.skel.h"
 #include "verifier_bpf_trap.skel.h"
 #include "verifier_bswap.skel.h"
+#include "verifier_btf_array_access.skel.h"
 #include "verifier_btf_ctx_access.skel.h"
 #include "verifier_btf_flex_array.skel.h"
 #include "verifier_btf_unreliable_prog.skel.h"
@@ -213,6 +214,7 @@ void test_verifier_bounds_mix_sign_unsign(void) { RUN(verifier_bounds_mix_sign_u
 void test_verifier_bpf_get_stack(void)        { RUN(verifier_bpf_get_stack); }
 void test_verifier_bpf_trap(void)             { RUN(verifier_bpf_trap); }
 void test_verifier_bswap(void)                { RUN(verifier_bswap); }
+void test_verifier_btf_array_access(void)     { RUN(verifier_btf_array_access); }
 void test_verifier_btf_ctx_access(void)       { RUN(verifier_btf_ctx_access); }
 void test_verifier_btf_flex_array(void)       { RUN(verifier_btf_flex_array); }
 void test_verifier_btf_unreliable_prog(void)  { RUN(verifier_btf_unreliable_prog); }
diff --git a/tools/testing/selftests/bpf/progs/verifier_btf_array_access.c b/tools/testing/selftests/bpf/progs/verifier_btf_array_access.c
new file mode 100644
index 000000000000..be001cc77f81
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/verifier_btf_array_access.c
@@ -0,0 +1,576 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_core_read.h>
+#include "bpf_experimental.h"
+#include "bpf_misc.h"
+
+/*
+ * Variable-offset access into arrays within a BTF access chain, e.g. a[i].b,
+ * where 'i' is a register with a varying offset. The register's step has to
+ * match the stride of one of the arrays crossed while walking the access
+ * chain, and the maximal possible offset has to stay within that array.
+ *
+ * A program-allocated object (bpf_obj_new) is used as the source of a
+ * PTR_TO_BTF_ID with a shape we fully control.
+ */
+struct inner {
+	int a;
+	int b;
+};
+
+struct outer {
+	struct inner arr[8];	/* off 0,   stride 8, size 64 */
+	char bytes[32];		/* off 64,  stride 1, size 32 */
+	long longs[8];		/* off 96,  stride 8, size 64 */
+	int grid[4][4];		/* off 160, 16 ints,  size 64 */
+};
+
+/* arr[i].b, i in [0, 7], 4-byte read: stays within arr. */
+SEC("syscall")
+__success
+int arr_field_in_bounds(void *ctx)
+{
+	unsigned long i = bpf_get_prandom_u32();
+	struct outer *o;
+	int val = 0;
+
+	o = bpf_obj_new(typeof(*o));
+	if (!o)
+		return 0;
+	asm volatile ("						\
+	r1 = %[o];						\
+	r2 = %[i];						\
+	r2 &= 7;						\
+	r2 *= 8;						\
+	r1 += r2;						\
+	%[val] = *(u32 *)(r1 + 4);				\
+"	: [val] "=r"(val)
+	: [o] "r"(o), [i] "r"(i)
+	: "r1", "r2");
+	bpf_obj_drop(o);
+	return val;
+}
+
+/* arr[i].b, i in [0, 15]: max offset runs past the end of arr. */
+SEC("syscall")
+__failure
+__msg("invalid variable offset access into struct outer")
+int arr_field_out_of_bounds(void *ctx)
+{
+	unsigned long i = bpf_get_prandom_u32();
+	struct outer *o;
+	int val = 0;
+
+	o = bpf_obj_new(typeof(*o));
+	if (!o)
+		return 0;
+	asm volatile ("						\
+	r1 = %[o];						\
+	r2 = %[i];						\
+	r2 &= 15;						\
+	r2 *= 8;						\
+	r1 += r2;						\
+	%[val] = *(u32 *)(r1 + 4);				\
+"	: [val] "=r"(val)
+	: [o] "r"(o), [i] "r"(i)
+	: "r1", "r2");
+	bpf_obj_drop(o);
+	return val;
+}
+
+/* longs[i], i in [0, 7], 8-byte read: scalar array with stride 8. */
+SEC("syscall")
+__success
+int long_array_in_bounds(void *ctx)
+{
+	unsigned long i = bpf_get_prandom_u32();
+	struct outer *o;
+	long val = 0;
+
+	o = bpf_obj_new(typeof(*o));
+	if (!o)
+		return 0;
+	asm volatile ("						\
+	r1 = %[o];						\
+	r2 = %[i];						\
+	r2 &= 7;						\
+	r2 *= 8;						\
+	r1 += r2;						\
+	%[val] = *(u64 *)(r1 + 96);				\
+"	: [val] "=r"(val)
+	: [o] "r"(o), [i] "r"(i)
+	: "r1", "r2");
+	bpf_obj_drop(o);
+	return val;
+}
+
+/*
+ * bytes[i], i in [0, 31], read as 4 bytes: min_off is valid within the array,
+ * but the 4-byte read at max_off runs past the array end. Exercises the
+ * size-aware bound in btf_struct_access().
+ */
+SEC("syscall")
+__failure
+__msg("invalid variable offset access into struct outer")
+int byte_array_size_spanning(void *ctx)
+{
+	unsigned long i = bpf_get_prandom_u32();
+	struct outer *o;
+	int val = 0;
+
+	o = bpf_obj_new(typeof(*o));
+	if (!o)
+		return 0;
+	asm volatile ("						\
+	r1 = %[o];						\
+	r2 = %[i];						\
+	r2 &= 31;						\
+	r1 += r2;						\
+	%[val] = *(u32 *)(r1 + 64);				\
+"	: [val] "=r"(val)
+	: [o] "r"(o), [i] "r"(i)
+	: "r1", "r2");
+	bpf_obj_drop(o);
+	return val;
+}
+
+/* bytes[i], i in [0, 31], 1-byte read: step 1 matches the char array stride. */
+SEC("syscall")
+__success
+int byte_array_in_bounds(void *ctx)
+{
+	unsigned long i = bpf_get_prandom_u32();
+	struct outer *o;
+	int val = 0;
+
+	o = bpf_obj_new(typeof(*o));
+	if (!o)
+		return 0;
+	asm volatile ("						\
+	r1 = %[o];						\
+	r2 = %[i];						\
+	r2 &= 31;						\
+	r1 += r2;						\
+	%[val] = *(u8 *)(r1 + 64);				\
+"	: [val] "=r"(val)
+	: [o] "r"(o), [i] "r"(i)
+	: "r1", "r2");
+	bpf_obj_drop(o);
+	return val;
+}
+
+/*
+ * arr[] accessed with step 4 (half of sizeof(struct inner)): the step is not
+ * a whole number of elements, so no crossed array has a matching stride.
+ */
+SEC("syscall")
+__failure
+__msg("invalid variable offset access into struct outer")
+int arr_unaligned_step(void *ctx)
+{
+	unsigned long i = bpf_get_prandom_u32();
+	struct outer *o;
+	int val = 0;
+
+	o = bpf_obj_new(typeof(*o));
+	if (!o)
+		return 0;
+	asm volatile ("						\
+	r1 = %[o];						\
+	r2 = %[i];						\
+	r2 &= 7;						\
+	r2 *= 4;						\
+	r1 += r2;						\
+	%[val] = *(u32 *)(r1 + 0);				\
+"	: [val] "=r"(val)
+	: [o] "r"(o), [i] "r"(i)
+	: "r1", "r2");
+	bpf_obj_drop(o);
+	return val;
+}
+
+/*
+ * grid[i][0], i in [0, 3]: steps along the outer dimension of a 2D array
+ * (stride 16). __btf_resolve_size() linearizes the array; the step is a
+ * multiple of the innermost element size (4).
+ */
+SEC("syscall")
+__success
+int grid_outer_dim(void *ctx)
+{
+	unsigned long i = bpf_get_prandom_u32();
+	struct outer *o;
+	int val = 0;
+
+	o = bpf_obj_new(typeof(*o));
+	if (!o)
+		return 0;
+	asm volatile ("						\
+	r1 = %[o];						\
+	r2 = %[i];						\
+	r2 &= 3;						\
+	r2 *= 16;						\
+	r1 += r2;						\
+	%[val] = *(u32 *)(r1 + 160);				\
+"	: [val] "=r"(val)
+	: [o] "r"(o), [i] "r"(i)
+	: "r1", "r2");
+	bpf_obj_drop(o);
+	return val;
+}
+
+/*
+ * grid[0][i], i in [0, 3]: steps along the inner dimension of a 2D array
+ * (stride 4, the innermost element size).
+ */
+SEC("syscall")
+__success
+int grid_inner_dim(void *ctx)
+{
+	unsigned long i = bpf_get_prandom_u32();
+	struct outer *o;
+	int val = 0;
+
+	o = bpf_obj_new(typeof(*o));
+	if (!o)
+		return 0;
+	asm volatile ("						\
+	r1 = %[o];						\
+	r2 = %[i];						\
+	r2 &= 3;						\
+	r2 *= 4;						\
+	r1 += r2;						\
+	%[val] = *(u32 *)(r1 + 160);				\
+"	: [val] "=r"(val)
+	: [o] "r"(o), [i] "r"(i)
+	: "r1", "r2");
+	bpf_obj_drop(o);
+	return val;
+}
+
+/*
+ * An array of pointers within a program-allocated object. Reading a pointer
+ * member of a local object yields a SCALAR_VALUE, but the varying offset still
+ * has to be validated against the array bounds.
+ */
+struct with_ptrs {
+	struct inner *parr[8];	/* off 0, stride 8 (sizeof ptr), size 64 */
+};
+
+/* parr[i], i in [0, 7], 8-byte read: stays within parr. */
+SEC("syscall")
+__success
+int ptr_array_in_bounds(void *ctx)
+{
+	unsigned long i = bpf_get_prandom_u32();
+	struct with_ptrs *o;
+	long val = 0;
+
+	o = bpf_obj_new(typeof(*o));
+	if (!o)
+		return 0;
+	asm volatile ("						\
+	r1 = %[o];						\
+	r2 = %[i];						\
+	r2 &= 7;						\
+	r2 *= 8;						\
+	r1 += r2;						\
+	%[val] = *(u64 *)(r1 + 0);				\
+"	: [val] "=r"(val)
+	: [o] "r"(o), [i] "r"(i)
+	: "r1", "r2");
+	bpf_obj_drop(o);
+	return val;
+}
+
+/*
+ * parr[i], i in [0, 15]: max offset runs past the end of parr. Without the
+ * variable-offset check on the WALK_PTR path this would be wrongly accepted.
+ */
+SEC("syscall")
+__failure
+__msg("invalid variable offset access into struct with_ptrs")
+int ptr_array_out_of_bounds(void *ctx)
+{
+	unsigned long i = bpf_get_prandom_u32();
+	struct with_ptrs *o;
+	long val = 0;
+
+	o = bpf_obj_new(typeof(*o));
+	if (!o)
+		return 0;
+	asm volatile ("						\
+	r1 = %[o];						\
+	r2 = %[i];						\
+	r2 &= 15;						\
+	r2 *= 8;						\
+	r1 += r2;						\
+	%[val] = *(u64 *)(r1 + 0);				\
+"	: [val] "=r"(val)
+	: [o] "r"(o), [i] "r"(i)
+	: "r1", "r2");
+	bpf_obj_drop(o);
+	return val;
+}
+
+/*
+ * The cases above all use bpf_obj_new() (PTR_TO_BTF_ID | MEM_ALLOC). Exercise
+ * the trusted kernel-BTF path too, using a task_struct from
+ * bpf_get_current_task_btf() and its embedded char comm[] array. The comm
+ * offset is folded into the pointer as a constant before the varying index.
+ */
+
+/* task->comm[i], i in [0, 15], 1-byte read: stays within comm. */
+SEC("syscall")
+__success
+int kernel_btf_array_in_bounds(void *ctx)
+{
+	unsigned long i = bpf_get_prandom_u32();
+	struct task_struct *task;
+	int val = 0;
+
+	task = bpf_get_current_task_btf();
+	asm volatile ("						\
+	r1 = %[task];						\
+	r1 += %[off];						\
+	r2 = %[i];						\
+	r2 &= 15;						\
+	r1 += r2;						\
+	%[val] = *(u8 *)(r1 + 0);				\
+"	: [val] "=r"(val)
+	: [task] "r"(task), [i] "r"(i),
+	  [off] "i"(offsetof(struct task_struct, comm))
+	: "r1", "r2");
+	return val;
+}
+
+/* task->comm[i], i in [0, 31]: max offset runs past comm. */
+SEC("syscall")
+__failure
+__msg("invalid variable offset access into struct task_struct")
+int kernel_btf_array_out_of_bounds(void *ctx)
+{
+	unsigned long i = bpf_get_prandom_u32();
+	struct task_struct *task;
+	int val = 0;
+
+	task = bpf_get_current_task_btf();
+	asm volatile ("						\
+	r1 = %[task];						\
+	r1 += %[off];						\
+	r2 = %[i];						\
+	r2 &= 31;						\
+	r1 += r2;						\
+	%[val] = *(u8 *)(r1 + 0);				\
+"	: [val] "=r"(val)
+	: [task] "r"(task), [i] "r"(i),
+	  [off] "i"(offsetof(struct task_struct, comm))
+	: "r1", "r2");
+	return val;
+}
+
+/*
+ * Array nested below a struct member at a non-zero offset. The walk descends
+ * through 'm' (resetting the running offset) before reaching 'arr', exercising
+ * the array_start = min_off - arrays[i].off telescoping across a WALK_STRUCT
+ * dive.
+ */
+struct mid {
+	struct inner arr[8];	/* stride 8, size 64 */
+};
+
+struct nest {
+	long pad;		/* off 0 */
+	struct mid m;		/* off 8 */
+};
+
+/* m.arr[i].b, i in [0, 7], 4-byte read: stays within m.arr. */
+SEC("syscall")
+__success
+int nested_struct_in_bounds(void *ctx)
+{
+	unsigned long i = bpf_get_prandom_u32();
+	struct nest *o;
+	int val = 0;
+
+	o = bpf_obj_new(typeof(*o));
+	if (!o)
+		return 0;
+	asm volatile ("						\
+	r1 = %[o];						\
+	r2 = %[i];						\
+	r2 &= 7;						\
+	r2 *= 8;						\
+	r1 += r2;						\
+	%[val] = *(u32 *)(r1 + 12);				\
+"	: [val] "=r"(val)
+	: [o] "r"(o), [i] "r"(i)
+	: "r1", "r2");
+	bpf_obj_drop(o);
+	return val;
+}
+
+/* m.arr[i].b, i in [0, 15]: max offset runs past the end of m.arr. */
+SEC("syscall")
+__failure
+__msg("invalid variable offset access into struct nest")
+int nested_struct_out_of_bounds(void *ctx)
+{
+	unsigned long i = bpf_get_prandom_u32();
+	struct nest *o;
+	int val = 0;
+
+	o = bpf_obj_new(typeof(*o));
+	if (!o)
+		return 0;
+	asm volatile ("						\
+	r1 = %[o];						\
+	r2 = %[i];						\
+	r2 &= 15;						\
+	r2 *= 8;						\
+	r1 += r2;						\
+	%[val] = *(u32 *)(r1 + 12);				\
+"	: [val] "=r"(val)
+	: [o] "r"(o), [i] "r"(i)
+	: "r1", "r2");
+	bpf_obj_drop(o);
+	return val;
+}
+
+/*
+ * A trailing flexible array member makes the struct tail an unbounded region,
+ * so a varying access into it has no upper bound to exceed (mirrors how
+ * unix_address.name[] is accessed via sun_path[i]).
+ *
+ * A custom (program) BTF struct can only be reached as a PTR_TO_BTF_ID through
+ * bpf_obj_new(), which does not allow flexible array members. So use a kernel
+ * type via an untrusted PTR_TO_BTF_ID (bpf_core_cast()). struct vring_used ends
+ * with a flexible array 'ring[]' of struct vring_used_elem { __virtio32 id, len; },
+ * mirroring a 'struct foo { int a; int b; }' flexible array.
+ */
+
+/* ring[i].len, i in [0, 63], 4-byte read: the flexible array has no upper bound. */
+SEC("syscall")
+__success
+int flex_array_field(void *ctx)
+{
+	unsigned long i = bpf_get_prandom_u32();
+	struct vring_used *o;
+	int val = 0;
+
+	o = bpf_core_cast(bpf_get_current_task_btf(), struct vring_used);
+	asm volatile ("						\
+	r1 = %[o];						\
+	r2 = %[i];						\
+	r2 &= 63;						\
+	r2 *= 8;						\
+	r1 += r2;						\
+	%[val] = *(u32 *)(r1 + 8);				\
+"	: [val] "=r"(val)
+	: [o] "r"(o), [i] "r"(i)
+	: "r1", "r2");
+	return val;
+}
+
+/*
+ * ring[] accessed with step 4 (half of sizeof(struct vring_used_elem)): the
+ * step is not a whole number of elements, so the flexible array stride does
+ * not match.
+ */
+SEC("syscall")
+__failure
+__msg("invalid variable offset access into struct vring_used")
+int flex_array_unaligned_step(void *ctx)
+{
+	unsigned long i = bpf_get_prandom_u32();
+	struct vring_used *o;
+	int val = 0;
+
+	o = bpf_core_cast(bpf_get_current_task_btf(), struct vring_used);
+	asm volatile ("						\
+	r1 = %[o];						\
+	r2 = %[i];						\
+	r2 &= 7;						\
+	r2 *= 4;						\
+	r1 += r2;						\
+	%[val] = *(u32 *)(r1 + 4);				\
+"	: [val] "=r"(val)
+	: [o] "r"(o), [i] "r"(i)
+	: "r1", "r2");
+	return val;
+}
+
+/*
+ * Access either name[0].sun_path[0] or one byte past sun_path.
+ * Step 108 does not match sizeof(struct sockaddr_un), which is 110.
+ * It matches the inner char array's stride, but exceeds its bounds.
+ * The flexible-tail exemption must not apply to that fixed array.
+ */
+SEC("syscall")
+__failure
+__msg("invalid variable offset access into struct unix_address")
+int flex_array_inner_array_out_of_bounds(void *ctx)
+{
+	unsigned long i = bpf_get_prandom_u32();
+	struct unix_address *o;
+	int val = 0;
+
+	o = bpf_core_cast(bpf_get_current_task_btf(), struct unix_address);
+	asm volatile ("						\
+	r1 = %[o];						\
+	r2 = %[i];						\
+	r2 &= 1;						\
+	r2 *= %[len];						\
+	r1 += r2;						\
+	%[val] = *(u8 *)(r1 + %[off]);				\
+"	: [val] "=r"(val)
+	: [o] "r"(o), [i] "r"(i),
+	  __imm_const(len, sizeof_field(struct sockaddr_un, sun_path)),
+	  __imm_const(off, offsetof(struct unix_address, name) +
+			   offsetof(struct sockaddr_un, sun_path))
+	: "r1", "r2");
+	return val;
+}
+
+typedef int array_element_t;
+typedef array_element_t array_type_t[4];
+
+struct typedef_array {
+	array_type_t values;
+};
+
+/*
+ * Resolving values' size crosses both the array and element typedefs.
+ * Record the array ID, not the int ID left by element-type resolution.
+ * values[i], i in [0, 3], must pass the variable-offset array check.
+ */
+SEC("syscall")
+__success
+int typedef_array_in_bounds(void *ctx)
+{
+	unsigned long i = bpf_get_prandom_u32();
+	struct typedef_array *o;
+	int val = 0;
+
+	o = bpf_obj_new(typeof(*o));
+	if (!o)
+		return 0;
+	asm volatile ("						\
+	r1 = %[o];						\
+	r2 = %[i];						\
+	r2 &= 3;						\
+	r2 *= %[elem_size];					\
+	r1 += r2;						\
+	%[val] = *(u32 *)(r1 + 0);				\
+"	: [val] "=r"(val)
+	: [o] "r"(o), [i] "r"(i),
+	  __imm_const(elem_size, sizeof(array_element_t))
+	: "r1", "r2");
+	bpf_obj_drop(o);
+	return val;
+}
+
+char _license[] SEC("license") = "GPL";

-- 
2.53.0

Re: [PATCH bpf-next v3 48/51] selftests/bpf: tests for varying offset access to PTR_TO_BTF_ID

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 2756 bytes --]

> selftests/bpf: tests for varying offset access to PTR_TO_BTF_ID

TL;DR: Test issue: byte_array_size_spanning may fail with a misaligned-access
message instead of the expected one on strict-alignment architectures
(riscv64, mips, loongarch); it may need __flag(BPF_F_ANY_ALIGNMENT).

> diff --git a/tools/testing/selftests/bpf/progs/verifier_btf_array_access.c b/tools/testing/selftests/bpf/progs/verifier_btf_array_access.c
> new file mode 100644
> index 000000000000..be001cc77f81
> --- /dev/null
> +++ b/tools/testing/selftests/bpf/progs/verifier_btf_array_access.c

[ ... ]

> +/*
> + * bytes[i], i in [0, 31], read as 4 bytes: min_off is valid within the array,
> + * but the 4-byte read at max_off runs past the array end. Exercises the
> + * size-aware bound in btf_struct_access().
> + */
> +SEC("syscall")
> +__failure
> +__msg("invalid variable offset access into struct outer")
> +int byte_array_size_spanning(void *ctx)
> +{
> +	unsigned long i = bpf_get_prandom_u32();
> +	struct outer *o;
> +	int val = 0;
> +
> +	o = bpf_obj_new(typeof(*o));
> +	if (!o)
> +		return 0;
> +	asm volatile ("						\
> +	r1 = %[o];						\
> +	r2 = %[i];						\
> +	r2 &= 31;						\
> +	r1 += r2;						\
> +	%[val] = *(u32 *)(r1 + 64);				\
> +"	: [val] "=r"(val)
> +	: [o] "r"(o), [i] "r"(i)
> +	: "r1", "r2");
> +	bpf_obj_drop(o);

byte_array_size_spanning() does a 4-byte load at a variable offset,
var_off (0x0; 0x1f) + 64, and expects btf_struct_access() to reject it with
"invalid variable offset access into struct outer".

check_mem_access() calls check_ptr_alignment() before it reaches
check_ptr_to_btf_access(), and bpf_check() turns on strict alignment when
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS is not set:

    env->strict_alignment = !!(attr->prog_flags & BPF_F_STRICT_ALIGNMENT);
    if (!IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS))
        env->strict_alignment = true;
    if (attr->prog_flags & BPF_F_ANY_ALIGNMENT)
        env->strict_alignment = false;

On such configurations (riscv64 with the default
RISCV_PROBE_UNALIGNED_ACCESS, mips, loongarch with ARCH_STRICT_ALIGN) the
load would be rejected first with:

    misaligned access off (0x0; 0x1f)+64 size 4

__failure would still hold, but __msg("invalid variable offset access into
struct outer") would not match.

Should this test add __flag(BPF_F_ANY_ALIGNMENT), as the negative tests in
verifier_array_access.c do when their expected message isn't about
alignment, so the size-aware bound is what gets checked on every
architecture?


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/37923187511

[PATCH bpf-next v3 49/51] selftests/bpf: tests for loop hierarchy computation

[Eduard Zingerman]

Test cases covering the following branches in bpf_compute_loops():
- Case B: simple backedge creating a loop (loop_single)
- Case B: two independent loops (loop_two_independent)
- Case B + D: nested loops where the inner header's loop_header points
  to the outer header (loop_nested)
- Case C: diamond CFG with no loops (fwd_edges_no_loop)
- Case D: sibling inner loops within one outer loop
  (loop_nested_siblings)
- Three levels of loop nesting (loop_three_levels)
- Loop with an if-else body containing forward branches
  (loop_with_if_else)
- Case E: An irreducible loop (loop_irreducible)
- A self-loop (loop_self)
- A test with sibling loops nested in outer loops
  (loop_nested_siblings_common_ancestors)
- A test case with outer loop's backedge originating
  from an inner loop (loop_outer_backedge_from_inner).

Cases B, C, D and E are described in loops.c:compute_loops_in_subprog().

Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 tools/testing/selftests/bpf/prog_tests/verifier.c  |   2 +
 .../selftests/bpf/progs/verifier_loop_hierarchy.c  | 346 +++++++++++++++++++++
 2 files changed, 348 insertions(+)

diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c
index 2f05a0ce829f..6b60d7720e2a 100644
--- a/tools/testing/selftests/bpf/prog_tests/verifier.c
+++ b/tools/testing/selftests/bpf/prog_tests/verifier.c
@@ -74,6 +74,7 @@
 #include "verifier_live_stack.skel.h"
 #include "verifier_liveness_exp.skel.h"
 #include "verifier_load_acquire.skel.h"
+#include "verifier_loop_hierarchy.skel.h"
 #include "verifier_loops1.skel.h"
 #include "verifier_lwt.skel.h"
 #include "verifier_map_in_map.skel.h"
@@ -261,6 +262,7 @@ void test_verifier_leak_ptr(void)             { RUN(verifier_leak_ptr); }
 void test_verifier_linked_scalars(void)       { RUN(verifier_linked_scalars); }
 void test_verifier_live_stack(void)           { RUN(verifier_live_stack); }
 void test_verifier_liveness_exp(void)         { RUN(verifier_liveness_exp); }
+void test_verifier_loop_hierarchy(void)       { RUN(verifier_loop_hierarchy); }
 void test_verifier_loops1(void)               { RUN(verifier_loops1); }
 void test_verifier_lwt(void)                  { RUN(verifier_lwt); }
 void test_verifier_map_in_map(void)           { RUN(verifier_map_in_map); }
diff --git a/tools/testing/selftests/bpf/progs/verifier_loop_hierarchy.c b/tools/testing/selftests/bpf/progs/verifier_loop_hierarchy.c
new file mode 100644
index 000000000000..0608e983d7fc
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/verifier_loop_hierarchy.c
@@ -0,0 +1,346 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+/*
+ * kernel/bpf/loops.c:compute_loops_in_subprog() distinguishes between
+ * the following cases:
+ * - B: backedge -> simple loop
+ * - C: cross edge to non-loop node -> no-op
+ * - D: edge to node whose header is in DFS path -> nested loop
+ * - E: edge to node whose header is NOT in DFS path -> irreducible
+ *
+ * Below test cases cover the above branches in various combinations.
+ */
+
+/* Case B: single bounded loop. */
+SEC("socket")
+__success
+__log_level(2)
+__msg("loop at 1{{$}}")
+__msg("Program dump")
+__msg("         -1   0: {{.*}} (b7) r0 = 0")
+__msg("  1       0   1: {{.*}} (07) r0 += 1")
+__msg("  1   1   1   2: {{.*}} (a5) if r0 < 0xa goto pc-2")
+__msg("          2   3: {{.*}} (95) exit")
+__naked void loop_single(void)
+{
+	asm volatile ("					\
+	r0 = 0;						\
+1:	r0 += 1;					\
+	if r0 < 10 goto 1b;				\
+	exit;						\
+"	::: __clobber_all);
+}
+
+/* Case B: two independent loops at the same nesting level. */
+SEC("socket")
+__success
+__log_level(2)
+__msg("loop at 1{{$}}")
+__msg("loop at 4{{$}}")
+__msg("Program dump")
+__msg("         -1   0: {{.*}} (b7) r0 = 0")
+__msg("  2       0   1: {{.*}} (07) r0 += 1")
+__msg("  2   1   1   2: {{.*}} (a5) if r0 < 0xa goto pc-2")
+__msg("          2   3: {{.*}} (b7) r1 = 0")
+__msg("  1       3   4: {{.*}} (07) r1 += 1")
+__msg("  1   4   4   5: {{.*}} (a5) if r1 < 0xa goto pc-2")
+__msg("          5   6: {{.*}} (95) exit")
+__naked void loop_two_independent(void)
+{
+	asm volatile ("					\
+	r0 = 0;						\
+1:	r0 += 1;					\
+	if r0 < 10 goto 1b;				\
+	r1 = 0;						\
+2:	r1 += 1;					\
+	if r1 < 10 goto 2b;				\
+	exit;						\
+"	::: __clobber_all);
+}
+
+/* Case B + D: nested loops. */
+SEC("socket")
+__success
+__log_level(2)
+__msg("loop at 1{{$}}")
+__msg("loop at 3, nested in 1")
+__msg("Program dump")
+__msg("         -1   0: {{.*}} (b7) r0 = 0")
+__msg("  1       0   1: {{.*}} (07) r0 += 1")			/* outer loop header */
+__msg("  1   1   1   2: {{.*}} (b7) r1 = 0")			/* outer loop insn */
+__msg("  1   1   2   3: {{.*}} (07) r1 += 1")			/* inner loop header */
+__msg("  1   3   3   4: {{.*}} (a5) if r1 < 0x5 goto pc-2")	/* inner loop insn */
+__msg("  1   1   4   5: {{.*}} (a5) if r0 < 0xa goto pc-5")	/* outer loop insn */
+__msg("          5   6: {{.*}} (95) exit")
+__naked void loop_nested(void)
+{
+	asm volatile ("					\
+	r0 = 0;						\
+1:	r0 += 1;					\
+	r1 = 0;						\
+2:	r1 += 1;					\
+	if r1 < 5 goto 2b;				\
+	if r0 < 10 goto 1b;				\
+	exit;						\
+"	::: __clobber_all);
+}
+
+/* Case C: forward edges, no loops. */
+SEC("socket")
+__success
+__log_level(2)
+__msg("Program dump")
+__msg("         -1   0: {{.*}} (85) call bpf_get_prandom_u32")
+__msg("          0   1: {{.*}} (25) if r0 > 0x0 goto pc+2")
+__msg("          1   2: {{.*}} (b7) r0 = 2")
+__msg("          2   3: {{.*}} (05) goto pc+1")
+__msg("          1   4: {{.*}} (b7) r0 = 3")
+__msg("          1   5: {{.*}} (95) exit")
+__naked void fwd_edges_no_loop(void)
+{
+	asm volatile ("					\
+	call %[bpf_get_prandom_u32];			\
+	if r0 > 0 goto 1f;				\
+	r0 = 2;						\
+	goto 2f;					\
+1:	r0 = 3;						\
+2:	exit;						\
+"	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+/* Case B + D: two sibling inner loops within one outer loop. */
+SEC("socket")
+__success
+__log_level(2)
+__msg("loop at 1{{$}}")
+__msg("loop at 3, nested in 1")
+__msg("loop at 6, nested in 1")
+__msg("Program dump")
+__msg("         -1   0: {{.*}} (b7) r0 = 0")
+__msg("  1       0   1: {{.*}} (07) r0 += 1")
+__msg("  1   1   1   2: {{.*}} (b7) r1 = 0")
+__msg("  1   1   2   3: {{.*}} (07) r1 += 1")
+__msg("  1   3   3   4: {{.*}} (a5) if r1 < 0x5 goto pc-2")
+__msg("  1   1   4   5: {{.*}} (b7) r2 = 0")
+__msg("  1   1   5   6: {{.*}} (07) r2 += 1")
+__msg("  1   6   6   7: {{.*}} (a5) if r2 < 0x5 goto pc-2")
+__msg("  1   1   7   8: {{.*}} (a5) if r0 < 0xa goto pc-8")
+__msg("          8   9: {{.*}} (95) exit")
+__naked void loop_nested_siblings(void)
+{
+	asm volatile ("					\
+	r0 = 0;						\
+1:	r0 += 1;					\
+	r1 = 0;						\
+2:	r1 += 1;					\
+	if r1 < 5 goto 2b;				\
+	r2 = 0;						\
+3:	r2 += 1;					\
+	if r2 < 5 goto 3b;				\
+	if r0 < 10 goto 1b;				\
+	exit;						\
+"	::: __clobber_all);
+}
+
+/* Three levels of nesting. */
+SEC("socket")
+__success
+__log_level(2)
+__msg("loop at 1{{$}}")
+__msg("loop at 3, nested in 1")
+__msg("loop at 5, nested in 3")
+__msg("Program dump")
+__msg("         -1   0: {{.*}} (b7) r0 = 0")
+__msg("  1       0   1: {{.*}} (07) r0 += 1")
+__msg("  1   1   1   2: {{.*}} (b7) r1 = 0")
+__msg("  1   1   2   3: {{.*}} (07) r1 += 1")
+__msg("  1   3   3   4: {{.*}} (b7) r2 = 0")
+__msg("  1   3   4   5: {{.*}} (07) r2 += 1")
+__msg("  1   5   5   6: {{.*}} (a5) if r2 < 0x3 goto pc-2")
+__msg("  1   3   6   7: {{.*}} (a5) if r1 < 0x5 goto pc-5")
+__msg("  1   1   7   8: {{.*}} (a5) if r0 < 0xa goto pc-8")
+__msg("          8   9: {{.*}} (95) exit")
+__naked void loop_three_levels(void)
+{
+	asm volatile ("					\
+	r0 = 0;						\
+1:	r0 += 1;					\
+	r1 = 0;						\
+2:	r1 += 1;					\
+	r2 = 0;						\
+3:	r2 += 1;					\
+	if r2 < 3 goto 3b;				\
+	if r1 < 5 goto 2b;				\
+	if r0 < 10 goto 1b;				\
+	exit;						\
+"	::: __clobber_all);
+}
+
+/* Loop with an if-else body (forward branch inside loop, Case D). */
+SEC("socket")
+__success
+__log_level(2)
+__msg("loop at 1")
+__msg("Program dump")
+__msg("         -1   0: {{.*}} (b7) r0 = 0")
+__msg("  1       0   1: {{.*}} (07) r0 += 1")
+__msg("  1   1   1   2: {{.*}} (bf) r1 = r0")
+__msg("  1   1   2   3: {{.*}} (25) if r1 > 0x5 goto pc+1")
+__msg("  1   1   3   4: {{.*}} (b7) r1 = 1")
+__msg("  1   1   3   5: {{.*}} (0f) r0 += r1")
+__msg("  1   1   5   6: {{.*}} (a5) if r0 < 0x64 goto pc-6")
+__msg("          6   7: {{.*}} (95) exit")
+__naked void loop_with_if_else(void)
+{
+	asm volatile ("					\
+	r0 = 0;						\
+1:	r0 += 1;					\
+	r1 = r0;					\
+	if r1 > 5 goto 2f;				\
+	r1 = 1;						\
+2:	r0 += r1;					\
+	if r0 < 100 goto 1b;				\
+	exit;						\
+"	::: __clobber_all);
+}
+
+/* Case E: irreducible loop. */
+SEC("socket")
+__success
+__log_level(2)
+__msg("loop at 3, irreducible")
+__msg("Program dump")
+__msg("         -1   0: {{.*}} (85) call bpf_get_prandom_u32#7")
+__msg("          0   1: {{.*}} (b7) r1 = 0")
+__msg("          1   2: {{.*}} (25) if r0 > 0x5 goto pc+2")
+__msg("  1       2   3: {{.*}} (b7) r1 = 1")
+__msg("  1   3   3   4: {{.*}} (05) goto pc+1")
+__msg("          2   5: {{.*}} (b7) r1 = 2")
+__msg("  1   3   2   6: {{.*}} (0f) r0 += r1")
+__msg("  1   3   6   7: {{.*}} (a5) if r0 < 0x10 goto pc-5")
+__msg("          7   8: {{.*}} (95) exit")
+__naked void loop_irreducible(void)
+{
+	asm volatile ("					\
+	call %[bpf_get_prandom_u32];			\
+	r1 = 0;						\
+	if r0 > 5 goto 2f;				\
+1:	r1 = 1;						\
+	goto 3f;					\
+2:	r1 = 2;						\
+3:	r0 += r1;					\
+	if r0 < 16 goto 1b;				\
+	exit;						\
+"	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+SEC("socket")
+__failure
+__log_level(2)
+__msg("loop at 1")
+__msg("infinite loop detected at insn 1")
+__naked void loop_self(void)
+{
+	asm volatile ("					\
+	call %[bpf_get_prandom_u32];			\
+1:	if r0 < 10 goto 1b;				\
+	exit;						\
+"	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+/*
+ * Loops at 5 and 7 are nested inside loops at 3 and 1.
+ * The edge 6 -> 7 exits loop at 5 and enters loop at 7.
+ * Check that it is logged only as an exit from loop at 5.
+ *
+ *  0: r0 = 0;
+ *     do {
+ *  1:     r0++;
+ *  2:     r1 = 0;
+ *         do {
+ *  3:         r1++;
+ *  4:         r2 = 0;
+ *             do {
+ *  5:             r2++;
+ *  6:         } while (r2 < 3);
+ *             do {
+ *  7:             r2--;
+ *  8:         } while (r2 != 0);
+ *  9:     } while (r1 < 4);
+ * 10: } while (r0 < 5);
+ * 11: return r0;
+ */
+SEC("socket")
+__success
+__log_level(2)
+__msg("loop at 1{{$}}")
+__msg_next("  backedge from 10, latch at 10")
+__msg_next("  exit from 10 to 11")
+__msg_next("loop at 3, nested in 1")
+__msg_next("  backedge from 9, latch at 9")
+__msg_next("  exit from 9 to 10")
+__msg_next("loop at 5, nested in 3")
+__msg_next("  backedge from 6, latch at 6")
+__msg_next("  exit from 6 to 7")
+__msg_next("loop at 7, nested in 3")
+__msg_next("  backedge from 8, latch at 8")
+__msg_next("  exit from 8 to 9")
+__naked void loop_nested_siblings_common_ancestors(void)
+{
+	asm volatile ("					\
+	r0 = 0;						\
+1:	r0 += 1;					\
+	r1 = 0;						\
+2:	r1 += 1;					\
+	r2 = 0;						\
+3:	r2 += 1;					\
+	if r2 < 3 goto 3b;				\
+4:	r2 += -1;					\
+	if r2 != 0 goto 4b;				\
+	if r1 < 4 goto 2b;				\
+	if r0 < 5 goto 1b;				\
+	exit;						\
+"	::: __clobber_all);
+}
+
+/*
+ * The outer loop backedge (4 -> 2) originates in an inner loop.
+ * For now, latch should not be identified in such cases.
+ */
+SEC("socket")
+__success
+__log_level(2)
+__msg("loop at 2{{$}}")
+__msg_next("  backedge from 4{{$}}")
+__msg_next("  exit from 5 to 7")
+__msg_next("loop at 3, nested in 2")
+__msg_next("  backedge from 6, latch at 5")
+__msg_next("  exit from 4 to 2")
+__msg_next("  exit from 5 to 7")
+__naked void loop_outer_backedge_from_inner(void)
+{
+	asm volatile ("					\
+	r6 = 99;					\
+	r2 = 2;						\
+2:	r6 += 1;			/* loop at 2 */	\
+3:	r2 += 1;			/* loop at 3 */	\
+4:	if r2 == 3 goto 2b;				\
+5:	if r6 > 100 goto 7f;				\
+	goto 3b;					\
+7:	r0 = 0;						\
+	exit;						\
+"	::: __clobber_all);
+}
+
+char _license[] SEC("license") = "GPL";

-- 
2.53.0

[PATCH bpf-next v3 50/51] selftests/bpf: tests for immediate dominator computation

[Eduard Zingerman]

Coverage:
- straight-line code with an ldimm64 in the middle;
- an asymmetric if-then-else diamond (unequal-depth idoms_intersect());
- a simple loop, a loop with an if-else body, and nested loops;
- a loop header with two back edges (three predecessors);
- an irreducible CFG (fixpoint convergence);
- immediate dominators across subprogram boundaries, including a callee
  whose entry instruction is itself a loop header;
- a self-loop;
- an indirect jump (gotox) with a two-entry jump table.

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 tools/testing/selftests/bpf/prog_tests/verifier.c  |   2 +
 tools/testing/selftests/bpf/progs/verifier_idoms.c | 377 +++++++++++++++++++++
 2 files changed, 379 insertions(+)

diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c
index 6b60d7720e2a..b45916e04fef 100644
--- a/tools/testing/selftests/bpf/prog_tests/verifier.c
+++ b/tools/testing/selftests/bpf/prog_tests/verifier.c
@@ -58,6 +58,7 @@
 #include "verifier_helper_packet_access.skel.h"
 #include "verifier_helper_restricted.skel.h"
 #include "verifier_helper_value_access.skel.h"
+#include "verifier_idoms.skel.h"
 #include "verifier_int_ptr.skel.h"
 #include "verifier_iterating_callbacks.skel.h"
 #include "verifier_jeq_infer_not_null.skel.h"
@@ -246,6 +247,7 @@ void test_verifier_helper_access_var_len(void) { RUN(verifier_helper_access_var_
 void test_verifier_helper_packet_access(void) { RUN(verifier_helper_packet_access); }
 void test_verifier_helper_restricted(void)    { RUN(verifier_helper_restricted); }
 void test_verifier_helper_value_access(void)  { RUN(verifier_helper_value_access); }
+void test_verifier_idoms(void)                { RUN(verifier_idoms); }
 void test_verifier_int_ptr(void)              { RUN(verifier_int_ptr); }
 void test_verifier_iterating_callbacks(void)  { RUN(verifier_iterating_callbacks); }
 void test_verifier_jeq_infer_not_null(void)   { RUN(verifier_jeq_infer_not_null); }
diff --git a/tools/testing/selftests/bpf/progs/verifier_idoms.c b/tools/testing/selftests/bpf/progs/verifier_idoms.c
new file mode 100644
index 000000000000..fdee6bced77b
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/verifier_idoms.c
@@ -0,0 +1,377 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+#include "../../../include/linux/filter.h"
+
+/*
+ * The immediate dominator is printed by log_program() as the numeric column
+ * immediately before the "<insn#>:" field of every "Program dump" line:
+ *
+ *   Program dump (scc? loop_header? idom insn#: live_regs_before):
+ *            -1   0: ....... (b7) r0 = 0     <- idom(0) = -1 (subprog entry)
+ *             0   1: ....... (07) r0 += 1    <- idom(1) = 0
+ *             ^^^^^^
+ *             idom insn#
+ */
+
+/* Straight-line code, with an ldimm64 in the middle. */
+SEC("socket")
+__success
+__log_level(2)
+__msg("Program dump")
+__msg("{{.*}} -1   0: {{.*}} (b7) r0 = 0")
+__msg("{{.*}}  0   1: {{.*}} (18) r1 = 0x1122334455667788")
+__msg("{{.*}}  1   3: {{.*}} (07) r0 += 1")
+__msg("{{.*}}  3   4: {{.*}} (0f) r0 += r1")
+__msg("{{.*}}  4   5: {{.*}} (95) exit")
+__naked void straight_line_ldimm64(void)
+{
+	asm volatile ("					\
+	r0 = 0;						\
+	r1 = 0x1122334455667788 ll;			\
+	r0 += 1;					\
+	r0 += r1;					\
+	exit;						\
+"	::: __clobber_all);
+}
+
+/*
+ * Asymmetric if-then-else diamond: the "then" arm is 5 instructions long, the
+ * "else" arm is a single instruction. The merge point (insn 8) is dominated by
+ * the branch (insn 1), not by either arm. This forces idoms_intersect() to walk
+ * the two predecessors up unequal postorder depths before they meet.
+ */
+SEC("socket")
+__success
+__log_level(2)
+__msg("Program dump")
+__msg("{{.*}} -1   0: {{.*}} (85) call bpf_get_prandom_u32#7")
+__msg("{{.*}}  0   1: {{.*}} (25) if r0 > 0x0 goto pc+5")
+__msg("{{.*}}  1   2: {{.*}} (b7) r1 = 1")
+__msg("{{.*}}  2   3: {{.*}} (b7) r1 = 2")
+__msg("{{.*}}  3   4: {{.*}} (b7) r1 = 3")
+__msg("{{.*}}  4   5: {{.*}} (b7) r1 = 4")
+__msg("{{.*}}  5   6: {{.*}} (05) goto pc+1")
+__msg("{{.*}}  1   7: {{.*}} (b7) r1 = 9")
+__msg("{{.*}}  1   8: {{.*}} (95) exit")
+__naked void asymmetric_diamond(void)
+{
+	asm volatile ("					\
+	call %[bpf_get_prandom_u32];			\
+	if r0 > 0 goto 1f;				\
+	r1 = 1;						\
+	r1 = 2;						\
+	r1 = 3;						\
+	r1 = 4;						\
+	goto 2f;					\
+1:	r1 = 9;						\
+2:	exit;						\
+"	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+/*
+ * Simple loop: the header (insn 1) is dominated by the pre-header (insn 0). The
+ * back edge (insn 2 -> insn 1) must not change idom(1); the back-edge
+ * predecessor folds up via idoms_intersect().
+ */
+SEC("socket")
+__success
+__log_level(2)
+__msg("Program dump")
+__msg("{{.*}} -1   0: {{.*}} (b7) r0 = 0")
+__msg("{{.*}}  0   1: {{.*}} (07) r0 += 1")
+__msg("{{.*}}  1   2: {{.*}} (a5) if r0 < 0xa goto pc-2")
+__msg("{{.*}}  2   3: {{.*}} (95) exit")
+__naked void simple_loop(void)
+{
+	asm volatile ("					\
+	r0 = 0;						\
+1:	r0 += 1;					\
+	if r0 < 10 goto 1b;				\
+	exit;						\
+"	::: __clobber_all);
+}
+
+/*
+ * Loop with an if-else in its body. The in-loop merge point (insn 5) is
+ * dominated by the in-loop branch (insn 3), combining a back-edge intersect
+ * with a forward-diamond intersect.
+ */
+SEC("socket")
+__success
+__log_level(2)
+__msg("Program dump")
+__msg("{{.*}} -1   0: {{.*}} (b7) r0 = 0")
+__msg("{{.*}}  0   1: {{.*}} (07) r0 += 1")
+__msg("{{.*}}  1   2: {{.*}} (bf) r1 = r0")
+__msg("{{.*}}  2   3: {{.*}} (25) if r1 > 0x5 goto pc+1")
+__msg("{{.*}}  3   4: {{.*}} (b7) r1 = 1")
+__msg("{{.*}}  3   5: {{.*}} (0f) r0 += r1")
+__msg("{{.*}}  5   6: {{.*}} (a5) if r0 < 0x64 goto pc-6")
+__msg("{{.*}}  6   7: {{.*}} (95) exit")
+__naked void loop_with_if_else_body(void)
+{
+	asm volatile ("					\
+	r0 = 0;						\
+1:	r0 += 1;					\
+	r1 = r0;					\
+	if r1 > 5 goto 2f;				\
+	r1 = 1;						\
+2:	r0 += r1;					\
+	if r0 < 100 goto 1b;				\
+	exit;						\
+"	::: __clobber_all);
+}
+
+/*
+ * Nested loops: the idom chain runs inner-header -> outer-body -> outer-header
+ * -> pre-header. Exercises intersect across back edges at two nesting depths.
+ */
+SEC("socket")
+__success
+__log_level(2)
+__msg("Program dump")
+__msg("{{.*}} -1   0: {{.*}} (b7) r0 = 0")
+__msg("{{.*}}  0   1: {{.*}} (07) r0 += 1")
+__msg("{{.*}}  1   2: {{.*}} (b7) r1 = 0")
+__msg("{{.*}}  2   3: {{.*}} (07) r1 += 1")
+__msg("{{.*}}  3   4: {{.*}} (a5) if r1 < 0x5 goto pc-2")
+__msg("{{.*}}  4   5: {{.*}} (a5) if r0 < 0xa goto pc-5")
+__msg("{{.*}}  5   6: {{.*}} (95) exit")
+__naked void nested_loops(void)
+{
+	asm volatile ("					\
+	r0 = 0;						\
+1:	r0 += 1;					\
+	r1 = 0;						\
+2:	r1 += 1;					\
+	if r1 < 5 goto 2b;				\
+	if r0 < 10 goto 1b;				\
+	exit;						\
+"	::: __clobber_all);
+}
+
+/*
+ * Loop header (insn 1) with two back edges (from insn 4 and insn 5), i.e. three
+ * predecessors. Both back edges route through the incrementing header, so the
+ * loop is bounded and verifies. Repeated idoms_intersect() at the header must
+ * stay stable and keep idom(1) at the pre-header (insn 0).
+ */
+SEC("socket")
+__success
+__log_level(2)
+__msg("Program dump")
+__msg("{{.*}} -1   0: {{.*}} (b7) r6 = 0")
+__msg("{{.*}}  0   1: {{.*}} (07) r6 += 1")
+__msg("{{.*}}  1   2: {{.*}} (25) if r6 > 0xa goto pc+3")
+__msg("{{.*}}  2   3: {{.*}} (bf) r7 = r6")
+__msg("{{.*}}  3   4: {{.*}} (a5) if r7 < 0x5 goto pc-4")
+__msg("{{.*}}  4   5: {{.*}} (05) goto pc-5")
+__msg("{{.*}}  2   6: {{.*}} (b7) r0 = 0")
+__msg("{{.*}}  6   7: {{.*}} (95) exit")
+__naked void multi_backedge_header(void)
+{
+	asm volatile ("					\
+	r6 = 0;						\
+1:	r6 += 1;					\
+	if r6 > 10 goto 2f;				\
+	r7 = r6;					\
+	if r7 < 5 goto 1b;				\
+	goto 1b;					\
+2:	r0 = 0;						\
+	exit;						\
+"	::: __clobber_all);
+}
+
+/*
+ * Irreducible CFG (loop with two entries, insn 3 and insn 5, reached from the
+ * insn 2 branch). Dominators remain well-defined; this is a convergence test
+ * for the fixpoint under irreducibility. Note insn 6 is dominated by the branch
+ * (insn 2), not by either arm.
+ */
+SEC("socket")
+__success
+__log_level(2)
+__msg("Program dump")
+__msg("{{.*}} -1   0: {{.*}} (85) call bpf_get_prandom_u32#7")
+__msg("{{.*}}  0   1: {{.*}} (b7) r1 = 0")
+__msg("{{.*}}  1   2: {{.*}} (25) if r0 > 0x5 goto pc+2")
+__msg("{{.*}}  2   3: {{.*}} (b7) r1 = 1")
+__msg("{{.*}}  3   4: {{.*}} (05) goto pc+1")
+__msg("{{.*}}  2   5: {{.*}} (b7) r1 = 2")
+__msg("{{.*}}  2   6: {{.*}} (0f) r0 += r1")
+__msg("{{.*}}  6   7: {{.*}} (a5) if r0 < 0x10 goto pc-5")
+__msg("{{.*}}  7   8: {{.*}} (95) exit")
+__naked void irreducible(void)
+{
+	asm volatile ("					\
+	call %[bpf_get_prandom_u32];			\
+	r1 = 0;						\
+	if r0 > 5 goto 2f;				\
+1:	r1 = 1;						\
+	goto 3f;					\
+2:	r1 = 2;						\
+3:	r0 += r1;					\
+	if r0 < 16 goto 1b;				\
+	exit;						\
+"	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+/*
+ * Idoms are computed per subprog. Layout (libbpf preorder) is:
+ *   main: 0..1, sub: 2..7
+ * The callee entry (insn 2) must have idom -1 (subprog reset), and idoms inside
+ * the callee must reference only the callee's instructions, never main's. The
+ * in-callee merge (insn 7) is dominated by the in-callee branch (insn 3). The
+ * branch is on a prandom value so it is not constant-folded away.
+ */
+static __naked __noinline __used
+unsigned long idoms_diamond_sub(void)
+{
+	asm volatile ("					\
+	call %[bpf_get_prandom_u32];			\
+	if r0 > 0 goto 1f;				\
+	r0 = 1;						\
+	goto 2f;					\
+1:	r0 = 2;						\
+2:	exit;						\
+"	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+SEC("socket")
+__success
+__log_level(2)
+__msg("Program dump")
+__msg("{{.*}} -1   0: {{.*}} (85) call pc+1")
+__msg("{{.*}}  0   1: {{.*}} (95) exit")
+__msg("{{.*}} -1   2: {{.*}} (85) call bpf_get_prandom_u32#7")
+__msg("{{.*}}  2   3: {{.*}} (25) if r0 > 0x0 goto pc+2")
+__msg("{{.*}}  3   4: {{.*}} (b7) r0 = 1")
+__msg("{{.*}}  4   5: {{.*}} (05) goto pc+1")
+__msg("{{.*}}  3   6: {{.*}} (b7) r0 = 2")
+__msg("{{.*}}  3   7: {{.*}} (95) exit")
+__naked void multi_subprog(void)
+{
+	asm volatile ("					\
+	call idoms_diamond_sub;				\
+	exit;						\
+"	::: __clobber_all);
+}
+
+/*
+ * The first instruction of a subprog (insn 3) is itself a loop header, i.e. it
+ * has an incoming back edge. Its idom must still be -1 (the subprog entry has no
+ * dominator). Exercises the idoms[start]=start ... idoms[start]=-1 handling in
+ * compute_subprog_idoms().
+ */
+static __naked __noinline __used
+unsigned long idoms_entry_header_sub(void)
+{
+	asm volatile ("					\
+1:	r1 += 1;					\
+	if r1 < 10 goto 1b;				\
+	r0 = 0;						\
+	exit;						\
+"	::: __clobber_all);
+}
+
+SEC("socket")
+__success
+__log_level(2)
+__msg("Program dump")
+__msg("{{.*}} -1   0: {{.*}} (b7) r1 = 0")
+__msg("{{.*}}  0   1: {{.*}} (85) call pc+1")
+__msg("{{.*}}  1   2: {{.*}} (95) exit")
+__msg("{{.*}} -1   3: {{.*}} (07) r1 += 1")
+__msg("{{.*}}  3   4: {{.*}} (a5) if r1 < 0xa goto pc-2")
+__msg("{{.*}}  4   5: {{.*}} (b7) r0 = 0")
+__msg("{{.*}}  5   6: {{.*}} (95) exit")
+__naked void entry_is_loop_header(void)
+{
+	asm volatile ("					\
+	r1 = 0;						\
+	call idoms_entry_header_sub;			\
+	exit;						\
+"	::: __clobber_all);
+}
+
+/*
+ * Self-loop: insn 1 is its own predecessor. idom(1) must be the pre-header
+ * (insn 0); the self-edge is ignored (idoms[pred] == -1 on the first pass, then
+ * idoms_intersect(a == b) short-circuits). Program is rejected later for an
+ * infinite loop, but the dump (and idoms) are printed before that.
+ */
+SEC("socket")
+__failure
+__log_level(2)
+__msg("Program dump")
+__msg("{{.*}} -1   0: {{.*}} (85) call bpf_get_prandom_u32#7")
+__msg("{{.*}}  0   1: {{.*}} (a5) if r0 < 0xa goto pc-1")
+__msg("{{.*}}  1   2: {{.*}} (95) exit")
+__msg("infinite loop detected")
+__naked void self_loop(void)
+{
+	asm volatile ("					\
+	call %[bpf_get_prandom_u32];			\
+1:	if r0 < 10 goto 1b;				\
+	exit;						\
+"	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+#if defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64) || defined(__TARGET_ARCH_powerpc)
+/*
+ * Indirect jump (gotox) with a two-entry jump table. The gotox (insn 4) has two
+ * successors (insn 5 and insn 7), so both targets have the gotox as their only
+ * predecessor and idom.
+ */
+SEC("socket")
+__success
+__log_level(2)
+__msg("Program dump")
+__msg("{{.*}} -1   0: {{.*}} (18) r0 = {{0x[0-9a-f]+}}")
+__msg("{{.*}}  0   2: {{.*}} (07) r0 += 8")
+__msg("{{.*}}  2   3: {{.*}} (79) r0 = *(u64 *)(r0 +0)")
+__msg("{{.*}}  3   4: {{.*}} (0d) gotox r0")
+__msg("{{.*}}  4   5: {{.*}} (b7) r0 = 0")
+__msg("{{.*}}  5   6: {{.*}} (95) exit")
+__msg("{{.*}}  4   7: {{.*}} (b7) r0 = 1")
+__msg("{{.*}}  7   8: {{.*}} (95) exit")
+__naked void gotox_jump_table(void)
+{
+	asm volatile ("						\
+	.pushsection .jumptables,\"\",@progbits;		\
+jt0_%=:								\
+	.quad ret0_%= - socket;					\
+	.quad ret1_%= - socket;					\
+	.size jt0_%=, 16;					\
+	.global jt0_%=;						\
+	.popsection;						\
+								\
+	r0 = jt0_%= ll;						\
+	r0 += 8;						\
+	r0 = *(u64 *)(r0 + 0);					\
+	.8byte %[gotox_r0];					\
+ret0_%=:							\
+	r0 = 0;							\
+	exit;							\
+ret1_%=:							\
+	r0 = 1;							\
+	exit;							\
+"	:
+	: __imm_insn(gotox_r0, BPF_RAW_INSN(BPF_JMP | BPF_JA | BPF_X, BPF_REG_0, 0, 0, 0))
+	: __clobber_all);
+}
+#endif
+
+char _license[] SEC("license") = "GPL";

-- 
2.53.0

[PATCH bpf-next v3 51/51] selftests/bpf: tests for SCEV analysis and loop widening

[Eduard Zingerman]

- Computed expression shapes: register and spilled-counter recurrences,
  substitution at latches, agreeing and conflicting backedge updates,
  and ALU, extension, byte-swap, load/store and atomic expressions.

- Iteration bounds and header-visit counts for:
  - pre-condition loops using JEQ, JLT, JLE, JGE and JGT;
  - post-condition loops using JLT, JLE, JGE, JGT and JNE;
  - signed pre- and post-conditions using JSLT, JSLE, JSGT and JSGE;
  - increasing and decreasing counters, nonzero initial values,
    immediate and invariant-register bounds, and immediate exits;
  - additional exits that reduce the minimum iteration count or
    tighten the upper bound;
  - rounded bounds, counter wraparound, count overflow, extreme
    steps and JNE divisibility.

- Widening, backedge clamping and state-pruning convergence,
  including multiple induction variables used for map-value accesses
  and nonconstant entry values requiring a common alignment.
  Check precision propagation to initial values and bounds.

- Nested and sibling loops, shared headers, exits to another loop
  header, exits across multiple nesting levels, and side entries
  into loop nests.
  Check conservative summaries for irreducible children while still
  widening supported reducible descendants, value preservation on
  inner-loop exits, and detection of nonterminating outer loops.

- Invalidation of stack-slot expressions after indirect stack writes
  and global calls. Widening across bpf_probe_read_kernel() calls
  in static subprograms.

- Stack spill widths and slot states, including narrow spills, zero,
  unallocated and STACK_MISC slots, and counters spilled and filled
  using store-release/load-acquire.

- Avoid widening registers used to address stack spills, fills and
  dynptr constructor arguments, including dependencies through nested
  loops. Conversely, allow widening addresses of byte-sized stores.

- Conditional assignments joining scalar values and stack pointers,
  including preservation of a common base and gcd-derived step.

- Pointer widening and comparisons with compatible memory origins,
  fallback for incompatible origins and nullable pointers, and
  invalidation of stale packet pointer relationships.

- Conservative fallback for unsupported recurrences, unavailable
  counter spills and zero computed header counts.

Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
---
 tools/testing/selftests/bpf/prog_tests/verifier.c |    2 +
 tools/testing/selftests/bpf/progs/verifier_scev.c | 3180 +++++++++++++++++++++
 2 files changed, 3182 insertions(+)

diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c
index b45916e04fef..c69c0f7e9cc8 100644
--- a/tools/testing/selftests/bpf/prog_tests/verifier.c
+++ b/tools/testing/selftests/bpf/prog_tests/verifier.c
@@ -108,6 +108,7 @@
 #include "verifier_ringbuf.skel.h"
 #include "verifier_runtime_jit.skel.h"
 #include "verifier_scalar_ids.skel.h"
+#include "verifier_scev.skel.h"
 #include "verifier_sdiv.skel.h"
 #include "verifier_search_pruning.skel.h"
 #include "verifier_sock.skel.h"
@@ -296,6 +297,7 @@ void test_verifier_regalloc(void)             { RUN(verifier_regalloc); }
 void test_verifier_ringbuf(void)              { RUN(verifier_ringbuf); }
 void test_verifier_runtime_jit(void)          { RUN(verifier_runtime_jit); }
 void test_verifier_scalar_ids(void)           { RUN(verifier_scalar_ids); }
+void test_verifier_scev(void)                 { RUN(verifier_scev); }
 void test_verifier_sdiv(void)                 { RUN(verifier_sdiv); }
 void test_verifier_search_pruning(void)       { RUN(verifier_search_pruning); }
 void test_verifier_sock(void)                 { RUN(verifier_sock); }
diff --git a/tools/testing/selftests/bpf/progs/verifier_scev.c b/tools/testing/selftests/bpf/progs/verifier_scev.c
new file mode 100644
index 000000000000..59c4b2864db0
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/verifier_scev.c
@@ -0,0 +1,3180 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+#include <linux/bpf.h>
+#include <stdbool.h>
+#include <bpf/bpf_helpers.h>
+#include "../../../include/linux/filter.h"
+#include "bpf_misc.h"
+#include "bpf_kfuncs.h"
+
+#define COND_TEST(name, _initial, _step, _bound, msg, body)                     \
+SEC("xdp")                                                                      \
+__success                                                                       \
+__log_level(2)                                                                  \
+__flag(BPF_F_TEST_STATE_FREQ)                                                   \
+__msg("loop header at {{[0-9]+}}, " msg)                                        \
+__naked void name(void)                                                         \
+{                                                                               \
+	asm volatile (                                                          \
+		"r6 = %[initial] ll;"                                           \
+		"r7 = %[bound] ll;"                                             \
+		"r9 = 0;"                                                       \
+		body                                                            \
+		"2: r0 = 0;"                                                    \
+		"exit;"                                                         \
+		:                                                               \
+		: __imm_const(initial, _initial),                               \
+		  __imm_const(step, _step),                                     \
+		  __imm_const(bound, _bound)                                    \
+		: __clobber_all);                                               \
+}
+
+/* Optional assembly runs before the tested condition, preserving its latch. */
+#define PRE__COND_TEST(name, op, initial, step, bound, msg, ...)                 \
+	COND_TEST(name, initial, step, bound, msg,                              \
+		"1: r8 = %[step] ll;" __VA_ARGS__                               \
+		"if r6 " op " r7 goto 2f;"                                      \
+		"r6 += r8;"                                                     \
+		"goto 1b;")
+
+#define POST_COND_TEST(name, op, initial, step, bound, msg, ...)                \
+	COND_TEST(name, initial, step, bound, msg,                              \
+		"1: r8 = %[step] ll;" __VA_ARGS__                               \
+		"r6 += r8;"                                                     \
+		"if r6 " op " r7 goto 1b;")
+
+/*
+ * Bound fallback verification of otherwise infinite or extremely long loops.
+ * JSET is not a linear condition, SCEV ignores it and the tested condition
+ * alone decides the iteration count.
+ */
+#define LIMIT_ITERATIONS "r9 += 1; if r9 & 8 goto 2f;"
+
+#define PRE__COND_TEST2(name, op, initial, step, bound, msg)                     \
+	PRE__COND_TEST(name, op, initial, step, bound, msg, LIMIT_ITERATIONS)
+
+#define ITER_U64_MAX (~0ULL)
+#define ITER_S64_MAX ((1ULL << 63) - 1)
+#define ITER_S64_MIN (1ULL << 63)
+#define ITER_U32_MAX ((1ULL << 32) - 1)
+
+struct map_val {
+	char foo[1024];
+};
+
+struct {
+	__uint(type, BPF_MAP_TYPE_HASH);
+	__uint(max_entries, 1);
+	__type(key, int);
+	__type(value, struct map_val);
+} map SEC(".maps");
+
+typeof(map) other_map SEC(".maps");
+
+struct {
+	__uint(type, BPF_MAP_TYPE_ARRAY);
+	__uint(max_entries, 1);
+	__type(key, int);
+	__type(value, __u64);
+} small_map SEC(".maps");
+
+struct {
+	__uint(type, BPF_MAP_TYPE_RINGBUF);
+	__uint(max_entries, 4096);
+} ringbuf SEC(".maps");
+
+struct {
+	__uint(type, BPF_MAP_TYPE_ARRAY_OF_MAPS);
+	__uint(max_entries, 1);
+	__type(key, int);
+	__type(value, int);
+	__array(values, struct {
+		__uint(type, BPF_MAP_TYPE_ARRAY);
+		__uint(map_flags, BPF_F_INNER_MAP);
+		__uint(max_entries, 1);
+		__type(key, int);
+		__type(value, struct map_val);
+	});
+} outer_map SEC(".maps");
+
+struct bpf_iter__bpf_map_elem {
+	struct bpf_iter_meta *meta;
+	struct bpf_map *map;
+	void *key;
+	void *value;
+};
+
+SEC("xdp")
+__success
+__log_level(2)
+__msg_next("scev at header 1:")
+__msg_next("  r0=(+ r0 1) / (linear r0 1)")
+__naked void simple_loop1(void)
+{
+	asm volatile ("					\
+	r0 = 0;						\
+loop_%=:						\
+	if r0 == 10 goto exit_%=;			\
+	r0 += 1;					\
+	goto loop_%=;					\
+exit_%=:						\
+	r0 = 0;						\
+	exit;						\
+"	::: __clobber_all);
+}
+
+SEC("xdp")
+__success
+__log_level(2)
+__msg("8: (7b) *(u64 *)(r1 +0) = r0     ; *fp-8 (+ *fp-8 1) -> ?")
+__naked void indirect_write_invalidates_scev(void)
+{
+	asm volatile ("					\
+	r0 = 0;						\
+	*(u64 *)(r10 - 8) = r0;				\
+loop_%=:						\
+	r0 = *(u64 *)(r10 - 8);				\
+	if r0 == 10 goto exit_%=;			\
+	r0 += 1;					\
+	*(u64 *)(r10 - 8) = r0;				\
+	r1 = r10;					\
+	r1 += -8;					\
+	*(u64 *)(r1 + 0) = r0;				\
+	goto loop_%=;					\
+exit_%=:						\
+	r0 = 0;						\
+	exit;						\
+"	::: __clobber_all);
+}
+
+/* Even a no-op global callee can be freplaced with one that writes to *p. */
+__noinline int global_noop_counter(__u64 *p)
+{
+	return 0;
+}
+
+SEC("xdp")
+__success
+__log_level(2)
+__msg("8: (85) call pc+{{[0-9]+}} {{.*}}; *fp-8 (+ *fp-8 1) -> ?")
+__naked void global_call_invalidates_scev(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"*(u64 *)(r10 - 8) = r0;"
+	"r6 = 0;"
+"1:"
+	"r0 = *(u64 *)(r10 - 8);"
+	"r0 += 1;"
+	"*(u64 *)(r10 - 8) = r0;"
+	"r1 = r10;"
+	"r1 += -8;"
+	"call global_noop_counter;"
+	"r6 += 1;"
+	"if r6 < 3 goto 1b;"
+	"r0 = 0;"
+	"exit;"
+	::: __clobber_all);
+}
+
+SEC("xdp")
+__success
+__log_level(2)
+__msg_next("scev at header 2:")
+__msg_next("  *fp-8=(+ *fp-8 1) / (linear *fp-8 1)")
+__msg_next(" scev at latch 3:")
+__msg_next("  r0=*fp-8 / (linear *fp-8 1)")
+__msg_next("  *fp-8=*fp-8 / (linear *fp-8 1)")
+__naked void simple_loop2(void)
+{
+	asm volatile ("					\
+	r0 = 0;						\
+	*(u64 *)(r10 - 8) = r0;				\
+loop_%=:						\
+	r0 = *(u64 *)(r10 - 8);				\
+	if r0 == 10 goto exit_%=;			\
+	r0 = *(u64 *)(r10 - 8);				\
+	r0 += 1;					\
+	*(u64 *)(r10 - 8) = r0;				\
+	goto loop_%=;					\
+exit_%=:						\
+	r0 = 0;						\
+	exit;						\
+"	::: __clobber_all);
+}
+
+SEC("xdp")
+__success
+__log_level(2)
+__msg_next("scev at header 2:")
+__msg_next("  r1=(+ r1 1) / (linear r1 1)")
+__naked void meet_agrees(void)
+{
+	asm volatile ("					\
+	call %[bpf_get_prandom_u32];			\
+	r1 = 0;						\
+1:							\
+	if r1 == 2 goto 3f;				\
+	if r0 == 7 goto 2f;				\
+	r1 += 1;					\
+	goto 1b;					\
+2:							\
+	r1 += 1;					\
+	goto 1b;					\
+3:							\
+	r0 = 0;						\
+	exit;						\
+"	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+SEC("xdp")
+__failure
+__log_level(2)
+__msg_next("scev at header 1:")
+__msg_next("  r6=(any r6 (+ r6 1)) / ?")
+__naked void meet_disagrees(void)
+{
+	asm volatile ("					\
+	r6 = 0;						\
+1:							\
+	if r6 == 2 goto 3f;				\
+	call %[bpf_get_prandom_u32];			\
+	if r0 == 7 goto 1b;				\
+	r6 += 1;					\
+	goto 1b;					\
+3:							\
+	r0 = 0;						\
+	exit;						\
+"	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+/* Both entry state values for r1 should be accounted for */
+SEC("xdp")
+__success
+__log_level(2)
+__flag(BPF_F_TEST_STATE_FREQ)
+__msg("scev at header 2:")
+__msg_next("  r0=(+ r0 1) / (linear r0 1)")
+__msg_next("  r1=42 / (any r1 42)")
+__msg_next(" scev at latch 2:")
+__msg_next("  r0=(+ r0 1) / (linear r0 1)")
+__msg_next("  r1=42 / (any r1 42)")
+__msg("loop header at 2, widening r0 to 0..10 step 1")
+__msg_next("loop header at 2, widening r1 to 7..42 step 1")
+__naked void entry_state_join(void)
+{
+	asm volatile ("					\
+	r0 = 0;						\
+	r1 = 7;						\
+1:	if r0 == 10 goto 2f;	/* r1 in [7,42] */	\
+	r0 += 1; 					\
+	r1 = 42;					\
+	goto 1b;					\
+2:	r0 = r1;					\
+	exit;						\
+"	::: __clobber_all);
+}
+
+SEC("xdp")
+__log_level(2)
+__msg_next("scev at header 1:")
+__msg_next("  r6=(bswap32 (bswap32 (zext32 (- (- (zext32 (+ (>>...) 1))))))) / ?")
+__naked void expr_chain(void)
+{
+	asm volatile ("					\
+	r6 = 0;						\
+1:							\
+	if r6 == 2 goto 2f;				\
+	r6 += 1;					\
+	r6 <<= 32;					\
+	r6 >>= 32;					\
+	w6 += 1;					\
+	r6 = -r6;					\
+	w6 = -w6;					\
+	r6 = bswap32 r6;				\
+	r6 = bswap32 r6;				\
+	goto 1b;					\
+2:							\
+	r0 = 0;						\
+	exit;						\
+"	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+#define ALU_OP(insn) "r1 = r6; " insn "; r0 += r1;"
+
+SEC("xdp")
+__success
+__log_level(2)
+__msg("r1 += 2 {{.*}}; r1 r6 -> (+ r6 2)")
+__msg("r1 += r7 {{.*}}; r1 r6 -> (+ r6 r7)")
+__msg("w1 += 2 {{.*}}; r1 r6 -> (zext32 (+ r6 2))")
+__msg("w1 += w7 {{.*}}; r1 r6 -> (zext32 (+ r6 r7))")
+__msg("r1 -= 2 {{.*}}; r1 r6 -> (- r6 2)")
+__msg("r1 -= r7 {{.*}}; r1 r6 -> (- r6 r7)")
+__msg("w1 -= 2 {{.*}}; r1 r6 -> (zext32 (- r6 2))")
+__msg("w1 -= w7 {{.*}}; r1 r6 -> (zext32 (- r6 r7))")
+__msg("r1 *= 2 {{.*}}; r1 r6 -> (* r6 2)")
+__msg("r1 *= r7 {{.*}}; r1 r6 -> (* r6 r7)")
+__msg("w1 *= 2 {{.*}}; r1 r6 -> (zext32 (* r6 2))")
+__msg("w1 *= w7 {{.*}}; r1 r6 -> (zext32 (* r6 r7))")
+__msg("r1 /= 2 {{.*}}; r1 r6 -> (/ r6 2)")
+__msg("r1 /= r7 {{.*}}; r1 r6 -> (/ r6 r7)")
+__msg("w1 /= 2 {{.*}}; r1 r6 -> (zext32 (/ r6 2))")
+__msg("w1 /= w7 {{.*}}; r1 r6 -> (zext32 (/ r6 r7))")
+__msg("r1 s/= 2 {{.*}}; r1 r6 -> (s/ r6 2)")
+__msg("r1 s/= r7 {{.*}}; r1 r6 -> (s/ r6 r7)")
+__msg("w1 s/= 2 {{.*}}; r1 r6 -> (zext32 (s/ r6 2))")
+__msg("w1 s/= w7 {{.*}}; r1 r6 -> (zext32 (s/ r6 r7))")
+__msg("r1 %= 2 {{.*}}; r1 r6 -> (% r6 2)")
+__msg("r1 %= r7 {{.*}}; r1 r6 -> (% r6 r7)")
+__msg("w1 %= 2 {{.*}}; r1 r6 -> (zext32 (% r6 2))")
+__msg("w1 %= w7 {{.*}}; r1 r6 -> (zext32 (% r6 r7))")
+__msg("r1 s%= 2 {{.*}}; r1 r6 -> (s% r6 2)")
+__msg("r1 s%= r7 {{.*}}; r1 r6 -> (s% r6 r7)")
+__msg("w1 s%= 2 {{.*}}; r1 r6 -> (zext32 (s% r6 2))")
+__msg("w1 s%= w7 {{.*}}; r1 r6 -> (zext32 (s% r6 r7))")
+__msg("r1 |= 2 {{.*}}; r1 r6 -> (| r6 2)")
+__msg("r1 |= r7 {{.*}}; r1 r6 -> (| r6 r7)")
+__msg("w1 |= 2 {{.*}}; r1 r6 -> (zext32 (| r6 2))")
+__msg("w1 |= w7 {{.*}}; r1 r6 -> (zext32 (| r6 r7))")
+__msg("r1 &= 2 {{.*}}; r1 r6 -> (& r6 2)")
+__msg("r1 &= r7 {{.*}}; r1 r6 -> (& r6 r7)")
+__msg("w1 &= 2 {{.*}}; r1 r6 -> (zext32 (& r6 2))")
+__msg("w1 &= w7 {{.*}}; r1 r6 -> (zext32 (& r6 r7))")
+__msg("r1 ^= 2 {{.*}}; r1 r6 -> (^ r6 2)")
+__msg("r1 ^= r7 {{.*}}; r1 r6 -> (^ r6 r7)")
+__msg("w1 ^= 2 {{.*}}; r1 r6 -> (zext32 (^ r6 2))")
+__msg("w1 ^= w7 {{.*}}; r1 r6 -> (zext32 (^ r6 r7))")
+__msg("r1 <<= 2 {{.*}}; r1 r6 -> (<< r6 2)")
+__msg("r1 <<= r7 {{.*}}; r1 r6 -> (<< r6 r7)")
+__msg("w1 <<= 2 {{.*}}; r1 r6 -> (zext32 (<< r6 2))")
+__msg("w1 <<= w7 {{.*}}; r1 r6 -> (zext32 (<< r6 r7))")
+__msg("r1 >>= 2 {{.*}}; r1 r6 -> (>> r6 2)")
+__msg("r1 >>= r7 {{.*}}; r1 r6 -> (>> r6 r7)")
+__msg("w1 >>= 2 {{.*}}; r1 r6 -> (zext32 (>> r6 2))")
+__msg("w1 >>= w7 {{.*}}; r1 r6 -> (zext32 (>> r6 r7))")
+__msg("r1 s>>= 2 {{.*}}; r1 r6 -> (s>> r6 2)")
+__msg("r1 s>>= r7 {{.*}}; r1 r6 -> (s>> r6 r7)")
+__msg("w1 s>>= 2 {{.*}}; r1 r6 -> (zext32 (s>> r6 2))")
+__msg("w1 s>>= w7 {{.*}}; r1 r6 -> (zext32 (s>> r6 r7))")
+__msg("r1 = -r1 {{.*}}; r1 r6 -> (- r6)")
+__msg("w1 = -w1 {{.*}}; r1 r6 -> (zext32 (- r6))")
+__msg("r1 = 42 {{.*}}; r1 r6 -> 42")
+__msg("w1 = 42 {{.*}}; r1 r6 -> (zext32 42)")
+__msg("r1 = r7 {{.*}}; r1 r6 -> r7")
+__msg("w1 = w7 {{.*}}; r1 r6 -> (zext32 r7)")
+__msg("r1 = (s8)r6 {{.*}}; r1 r6 -> (sext8 r6)")
+__msg("r1 = (s16)r6 {{.*}}; r1 r6 -> (sext16 r6)")
+__msg("r1 = (s32)r6 {{.*}}; r1 r6 -> (sext32 r6)")
+__msg("w1 = (s8)w6 {{.*}}; r1 r6 -> (zext32 (sext8 r6))")
+__msg("w1 = (s16)w6 {{.*}}; r1 r6 -> (zext32 (sext16 r6))")
+__msg("r1 = bswap16 r1 {{.*}}; r1 r6 -> (bswap16 r6)")
+__msg("r1 = bswap32 r1 {{.*}}; r1 r6 -> (bswap32 r6)")
+__msg("r1 = bswap64 r1 {{.*}}; r1 r6 -> (bswap64 r6)")
+#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
+__msg("r1 = le16 r1 {{.*}}; r1 r6 -> (zext16 r6)")
+__msg("r1 = le32 r1 {{.*}}; r1 r6 -> (zext32 r6)")
+__msg("r1 += 1 {{.*}}; r1 r6 -> (+ r6 1)")
+__msg("r1 = be16 r1 {{.*}}; r1 r6 -> (bswap16 r6)")
+__msg("r1 = be32 r1 {{.*}}; r1 r6 -> (bswap32 r6)")
+__msg("r1 = be64 r1 {{.*}}; r1 r6 -> (bswap64 r6)")
+#else
+__msg("r1 = le16 r1 {{.*}}; r1 r6 -> (bswap16 r6)")
+__msg("r1 = le32 r1 {{.*}}; r1 r6 -> (bswap32 r6)")
+__msg("r1 = le64 r1 {{.*}}; r1 r6 -> (bswap64 r6)")
+__msg("r1 = be16 r1 {{.*}}; r1 r6 -> (zext16 r6)")
+__msg("r1 = be32 r1 {{.*}}; r1 r6 -> (zext32 r6)")
+__msg("r1 += 1 {{.*}}; r1 r6 -> (+ r6 1)")
+#endif
+__naked void alu_ops_exprs(void)
+{
+	asm volatile (
+	"r6 = 7;"
+	"r7 = 2;"
+	"r8 = 0;"
+"1:"
+	"r0 = 0;"
+	ALU_OP("r1 += 2")
+	ALU_OP("r1 += r7")
+	ALU_OP("w1 += 2")
+	ALU_OP("w1 += w7")
+	ALU_OP("r1 -= 2")
+	ALU_OP("r1 -= r7")
+	ALU_OP("w1 -= 2")
+	ALU_OP("w1 -= w7")
+	ALU_OP("r1 *= 2")
+	ALU_OP("r1 *= r7")
+	ALU_OP("w1 *= 2")
+	ALU_OP("w1 *= w7")
+	ALU_OP("r1 /= 2")
+	ALU_OP("r1 /= r7")
+	ALU_OP("w1 /= 2")
+	ALU_OP("w1 /= w7")
+	ALU_OP("r1 s/= 2")
+	ALU_OP("r1 s/= r7")
+	ALU_OP("w1 s/= 2")
+	ALU_OP("w1 s/= w7")
+	ALU_OP("r1 %%= 2")
+	ALU_OP("r1 %%= r7")
+	ALU_OP("w1 %%= 2")
+	ALU_OP("w1 %%= w7")
+	ALU_OP("r1 s%%= 2")
+	ALU_OP("r1 s%%= r7")
+	ALU_OP("w1 s%%= 2")
+	ALU_OP("w1 s%%= w7")
+	ALU_OP("r1 |= 2")
+	ALU_OP("r1 |= r7")
+	ALU_OP("w1 |= 2")
+	ALU_OP("w1 |= w7")
+	ALU_OP("r1 &= 2")
+	ALU_OP("r1 &= r7")
+	ALU_OP("w1 &= 2")
+	ALU_OP("w1 &= w7")
+	ALU_OP("r1 ^= 2")
+	ALU_OP("r1 ^= r7")
+	ALU_OP("w1 ^= 2")
+	ALU_OP("w1 ^= w7")
+	ALU_OP("r1 <<= 2")
+	ALU_OP("r1 <<= r7")
+	ALU_OP("w1 <<= 2")
+	ALU_OP("w1 <<= w7")
+	ALU_OP("r1 >>= 2")
+	ALU_OP("r1 >>= r7")
+	ALU_OP("w1 >>= 2")
+	ALU_OP("w1 >>= w7")
+	ALU_OP("r1 s>>= 2")
+	ALU_OP("r1 s>>= r7")
+	ALU_OP("w1 s>>= 2")
+	ALU_OP("w1 s>>= w7")
+	ALU_OP("r1 = -r1")
+	ALU_OP("w1 = -w1")
+	ALU_OP("r1 = 42")
+	ALU_OP("w1 = 42")
+	ALU_OP("r1 = r7")
+	ALU_OP("w1 = w7")
+	ALU_OP("r1 = (s8)r6")
+	ALU_OP("r1 = (s16)r6")
+	ALU_OP("r1 = (s32)r6")
+	ALU_OP("w1 = (s8)w6")
+	ALU_OP("w1 = (s16)w6")
+	ALU_OP("r1 = bswap16 r1")
+	ALU_OP("r1 = bswap32 r1")
+	ALU_OP("r1 = bswap64 r1")
+	ALU_OP("r1 = le16 r1")
+	ALU_OP("r1 = le32 r1")
+	/*
+	 * Expressions unchanged by transfer() do not appear in the log.
+	 * Add r1 += 1 to touch the expression and make it appear in the log.
+	 */
+	ALU_OP("r1 = le64 r1; r1 += 1")
+	ALU_OP("r1 = be16 r1")
+	ALU_OP("r1 = be32 r1")
+	ALU_OP("r1 = be64 r1; r1 += 1")
+	"r8 += 1;"
+	"if r8 < 2 goto 1b;"
+	"r0 = 0;"
+	"exit;"
+	::: __clobber_all);
+}
+
+#undef ALU_OP
+
+/* Reset operands and consume results to keep transfer changes live. */
+#define ST_OP(stmt)							\
+	"r0 = 0; r1 = 1; r2 = 2;"					\
+	"*(u64 *)(r10 - 8) = -8;"					\
+	stmt ";"							\
+	"r0 += r1; r0 += r2;"						\
+	"r3 = *(u64 *)(r10 - 8); r0 += r3;"
+
+SEC("xdp")
+__success
+__log_level(2)
+__msg("*(u64 *)(r10 -8) = r1 {{.*}}; *fp-8 -8 -> 1")
+__msg("*(u32 *)(r10 -8) = r1 {{.*}}; *fp-8 -8 -> (spill32 1)")
+__msg("*(u16 *)(r10 -8) = r1 {{.*}}; *fp-8 -8 -> (spill16 1)")
+__msg("*(u8 *)(r10 -8) = r1 {{.*}}; *fp-8 -8 -> (spill8 1)")
+__msg("r1 = *(u64 *)(r10 -8) {{.*}}; r1 1 -> -8")
+#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
+__msg("r1 = *(u32 *)(r10 -8) {{.*}}; r1 1 -> (zext32 -8)")
+__msg("r1 = *(u16 *)(r10 -8) {{.*}}; r1 1 -> (zext16 -8)")
+__msg("r1 = *(u8 *)(r10 -8) {{.*}}; r1 1 -> (zext8 -8)")
+__msg("r1 = *(s32 *)(r10 -8) {{.*}}; r1 1 -> (sext32 -8)")
+__msg("r1 = *(s16 *)(r10 -8) {{.*}}; r1 1 -> (sext16 -8)")
+__msg("r1 = *(s8 *)(r10 -8) {{.*}}; r1 1 -> (sext8 -8)")
+__msg("r1 = load_acquire((u8 *)(r10 -8)) {{.*}}; r1 1 -> (zext8 -8)")
+__msg("r1 = load_acquire((u16 *)(r10 -8)) {{.*}}; r1 1 -> (zext16 -8)")
+__msg("r1 = load_acquire((u32 *)(r10 -8)) {{.*}}; r1 1 -> (zext32 -8)")
+#else
+__msg("r1 = *(u32 *)(r10 -8) {{.*}}; r1 1 -> _")
+__msg("r1 = *(u16 *)(r10 -8) {{.*}}; r1 1 -> _")
+__msg("r1 = *(u8 *)(r10 -8) {{.*}}; r1 1 -> _")
+__msg("r1 = *(s32 *)(r10 -8) {{.*}}; r1 1 -> _")
+__msg("r1 = *(s16 *)(r10 -8) {{.*}}; r1 1 -> _")
+__msg("r1 = *(s8 *)(r10 -8) {{.*}}; r1 1 -> _")
+__msg("r1 = load_acquire((u8 *)(r10 -8)) {{.*}}; r1 1 -> _")
+__msg("r1 = load_acquire((u16 *)(r10 -8)) {{.*}}; r1 1 -> _")
+__msg("r1 = load_acquire((u32 *)(r10 -8)) {{.*}}; r1 1 -> _")
+#endif
+__msg("r2 = load_acquire((u64 *)(r10 -8)) {{.*}}; r2 2 -> -8")
+__msg("store_release((u8 *)(r10 -8), r1) {{.*}}; *fp-8 -8 -> (spill8 1)")
+__msg("store_release((u16 *)(r10 -8), r1) {{.*}}; *fp-8 -8 -> (spill16 1)")
+__msg("store_release((u32 *)(r10 -8), r1) {{.*}}; *fp-8 -8 -> (spill32 1)")
+__msg("store_release((u64 *)(r10 -8), r1) {{.*}}; *fp-8 -8 -> 1")
+__msg("lock *(u64 *)(r10 -8) += r1 {{.*}}; *fp-8 -8 -> _")
+__msg("r1 = atomic64_fetch_add((u64 *)(r10 -8), r1) {{.*}}; r1 1 -> _, *fp-8 -8 -> _")
+__msg("r1 = atomic_fetch_add((u32 *)(r10 -8), r1) {{.*}}; r1 1 -> _, *fp-8 -8 -> _")
+__msg("r1 = atomic64_xchg((u64 *)(r10 -8), r1) {{.*}}; r1 1 -> _, *fp-8 -8 -> _")
+__msg("r0 = atomic64_cmpxchg((u64 *)(r10 -8), r0, r2) {{.*}}; r0 0 -> _, *fp-8 -8 -> _")
+__msg("*(u64 *)(r10 -8) = 42 {{.*}}; *fp-8 -8 -> 42")
+__msg("*(u8 *)(r10 -7) = r0 {{.*}}; *fp-8 -8 -> _")
+__naked void store_load_exprs(void)
+{
+	asm volatile (
+	"r7 = 0;"
+"1:"
+	ST_OP("*(u64 *)(r10 - 8) = r1")
+	ST_OP("*(u32 *)(r10 - 8) = r1")
+	ST_OP("*(u16 *)(r10 - 8) = r1")
+	ST_OP("*(u8 *)(r10 - 8) = r1")
+
+	ST_OP("r1 = *(u64 *)(r10 - 8)")
+	ST_OP("r1 = *(u32 *)(r10 - 8)")
+	ST_OP("r1 = *(u16 *)(r10 - 8)")
+	ST_OP("r1 = *(u8 *)(r10 - 8)")
+	ST_OP("r1 = *(s32 *)(r10 - 8)")
+	ST_OP("r1 = *(s16 *)(r10 - 8)")
+	ST_OP("r1 = *(s8 *)(r10 - 8)")
+
+	ST_OP(".8byte %[load_acquire8]")
+	ST_OP(".8byte %[load_acquire16]")
+	ST_OP(".8byte %[load_acquire32]")
+	ST_OP(".8byte %[load_acquire64]")
+	ST_OP(".8byte %[store_release8]")
+	ST_OP(".8byte %[store_release16]")
+	ST_OP(".8byte %[store_release32]")
+	ST_OP(".8byte %[store_release64]")
+
+	ST_OP(".8byte %[atomic_add64]")
+	ST_OP(".8byte %[atomic_fetch_add64]")
+	ST_OP(".8byte %[atomic_fetch_add32]")
+	ST_OP(".8byte %[atomic_xchg64]")
+	ST_OP(".8byte %[atomic_cmpxchg64]")
+
+	ST_OP("*(u64 *)(r10 - 8) = 42")
+	ST_OP("*(u8 *)(r10 - 7) = r0")
+
+	"r7 += 1;"
+	"if r7 < 2 goto 1b;"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm_insn(atomic_add64,       BPF_ATOMIC_OP(BPF_DW, BPF_ADD,       BPF_REG_10, BPF_REG_1,  -8)),
+	  __imm_insn(atomic_xchg64,      BPF_ATOMIC_OP(BPF_DW, BPF_XCHG,      BPF_REG_10, BPF_REG_1,  -8)),
+	  __imm_insn(load_acquire8,      BPF_ATOMIC_OP(BPF_B,  BPF_LOAD_ACQ,  BPF_REG_1,  BPF_REG_10, -8)),
+	  __imm_insn(load_acquire16,     BPF_ATOMIC_OP(BPF_H,  BPF_LOAD_ACQ,  BPF_REG_1,  BPF_REG_10, -8)),
+	  __imm_insn(load_acquire32,     BPF_ATOMIC_OP(BPF_W,  BPF_LOAD_ACQ,  BPF_REG_1,  BPF_REG_10, -8)),
+	  __imm_insn(load_acquire64,     BPF_ATOMIC_OP(BPF_DW, BPF_LOAD_ACQ,  BPF_REG_2,  BPF_REG_10, -8)),
+	  __imm_insn(store_release8,     BPF_ATOMIC_OP(BPF_B,  BPF_STORE_REL, BPF_REG_10, BPF_REG_1,  -8)),
+	  __imm_insn(store_release16,    BPF_ATOMIC_OP(BPF_H,  BPF_STORE_REL, BPF_REG_10, BPF_REG_1,  -8)),
+	  __imm_insn(store_release32,    BPF_ATOMIC_OP(BPF_W,  BPF_STORE_REL, BPF_REG_10, BPF_REG_1,  -8)),
+	  __imm_insn(store_release64,    BPF_ATOMIC_OP(BPF_DW, BPF_STORE_REL, BPF_REG_10, BPF_REG_1,  -8)),
+	  __imm_insn(atomic_cmpxchg64,   BPF_ATOMIC_OP(BPF_DW, BPF_CMPXCHG,   BPF_REG_10, BPF_REG_2,  -8)),
+	  __imm_insn(atomic_fetch_add32, BPF_ATOMIC_OP(BPF_W,  BPF_ADD | BPF_FETCH, BPF_REG_10, BPF_REG_1, -8)),
+	  __imm_insn(atomic_fetch_add64, BPF_ATOMIC_OP(BPF_DW, BPF_ADD | BPF_FETCH, BPF_REG_10, BPF_REG_1, -8))
+	: __clobber_all);
+}
+
+#undef ST_OP
+
+SEC("xdp")
+__success
+__log_level(2)
+__msg_next("scev at header 1:")
+__msg_next("  r6=(+ r6 1) / (linear r6 1)")
+__msg_next("  r7=?")
+__msg_next(" scev at latch 1:")
+__msg_next("  r6=(+ r6 1) / (linear r6 1)")
+__msg_next("  r7=?")
+__msg_next("scev at header 4:")
+__msg_next("  r7=(+ r7 1) / (linear r7 1)")
+__msg_next(" scev at latch 4:")
+__msg_next("  r7=(+ r7 1) / (linear r7 1)")
+__naked void nested_loop1(void)
+{
+	asm volatile ("					\
+	r6 = 0;						\
+1:							\
+	if r6 == 2 goto 2f;				\
+	r6 += 1;					\
+	r7 = 0;						\
+3:							\
+	if r7 == 2 goto 4f;				\
+	r7 += 1;					\
+	goto 3b;					\
+4:							\
+	goto 1b;					\
+2:							\
+	r0 = r7;					\
+	exit;						\
+"	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+SEC("xdp")
+__success
+__msg("loop at 1")
+__msg_next("  backedge from 4{{$}}")
+__msg_next("  exit from 1 to 7")
+__msg_next("loop at 4, nested in 1")
+__msg_next("  backedge from 6, latch at 4")
+__msg_next("  exit from 4 to 1")
+__msg("scev at header 1:")
+__msg_next("  r6=(+ r6 1) / (linear r6 1)")
+__msg_next("  r7=?")
+__msg_next("scev at header 4:")
+__msg_next("  r7=(+ r7 1) / (linear r7 1)")
+__msg_next(" scev at latch 4:")
+__msg_next("  r7=(+ r7 1) / (linear r7 1)")
+__msg("loop header at 1, latch not identified")
+__msg("loop header at 4, widening r7 to 0..2 step 1")
+__log_level(2)
+__naked void nested_loop_hdr_backedge1(void)
+{
+	asm volatile ("					\
+	r6 = 0;						\
+1:							\
+	if r6 == 2 goto 3f;				\
+	r6 += 1;					\
+	r7 = 0;						\
+2:							\
+	  if r7 == 2 goto 1b;				\
+	  r7 += 1;					\
+	  goto 2b;					\
+3:							\
+	r0 = r7;					\
+	exit;						\
+"	::: __clobber_all);
+}
+
+/*
+ * Outer loop {2, 3, 4, 5} contains inner loop {3, 4, 5}.
+ * The outer loop's only backedge is 4 -> 2, and its latch is also at 4,
+ * inside the inner loop. Outer loop's latch index is not computed in
+ * such a case, as there is no mechanism to build SCEVs for such latches
+ * at the moment.
+ */
+SEC("xdp")
+__success
+__log_level(2)
+__flag(BPF_F_TEST_STATE_FREQ)
+__msg("loop at 2{{$}}")
+__msg_next("  backedge from 4{{$}}")
+__msg_next("  exit from 5 to 6")
+__msg_next("loop at 3, nested in 2")
+__msg_next("  backedge from 5, latch at 5")
+__msg_next("  exit from 4 to 2")
+__msg_next("  exit from 5 to 6")
+__msg("scev at header 2:")
+__msg_next("  r6=?")
+__msg_next("  r7=(+ r7 1) / (linear r7 1)")
+__msg_next("scev at header 3:")
+__msg_next("  r6=(+ r6 1) / (linear r6 1)")
+__msg_next(" scev at latch 5:")
+__msg_next("  r6=(+ r6 1) / (linear (+ r6 1) 1)")
+__msg("loop header at 2, latch not identified")
+__msg("loop header at 3, header_count is [0..2] ")
+__naked void nested_loop_hdr_backedge2(void)
+{
+	asm volatile ("					\
+	r6 = 0;						\
+	r7 = 0;						\
+1:	r7 += 1;					\
+2:	r6 += 1;					\
+	if r7 == 2 goto 1b;				\
+	if r6 < 2 goto 2b;				\
+	r0 = 0;						\
+	exit;						\
+"	::: __clobber_all);
+}
+
+/*
+ * Logical nest: for (r6 = 0; r6 < 3; r6++)
+ *                   for (r7 = 0; r7 < 4; r7++) r8++;
+ * Both backedges target header 3, so the CFG has one loop. r6 advances only
+ * on the outer backedge and r7 resets there; neither has a linear SCEV.
+ * r8 advances on both backedges and retains its linear SCEV.
+ */
+SEC("socket")
+__success
+__log_level(2)
+__flag(BPF_F_TEST_STATE_FREQ)
+__msg("loop at 3{{$}}")
+__msg("scev at header 3:")
+__msg_next("  r6=(any r6 (+ r6 1)) / ?")
+__msg_next("  r7=(any (+ r7 1) 0) / ?")
+__msg_next("  r8=(+ r8 1) / (linear r8 1)")
+__msg("loop header at 3, unsupported loop: multiple backedges")
+__naked void nested_loops_shared_header(void)
+{
+	asm volatile ("					\
+	r6 = 0;						\
+	r7 = 0;						\
+	r8 = 0;						\
+1:	r8 += 1;					\
+	r7 += 1;					\
+	if r7 < 4 goto 1b;				\
+	r7 = 0;						\
+	r6 += 1;					\
+	if r6 < 3 goto 1b;				\
+	r0 = r8;					\
+	exit;						\
+"	::: __clobber_all);
+}
+
+SEC("socket")
+__success
+__log_level(2)
+__msg("loop header at 1, header_count is 10")
+__msg("loop header at 5, header_count is 5")
+__flag(BPF_F_TEST_STATE_FREQ)
+__naked void nested_loops_outer_latch_true_branch_to_inner(void)
+{
+	asm volatile ("					\
+	r0 = 0;						\
+1:	r0 += 1;		/* outer loop */	\
+	r1 = 0;						\
+	if r0 < 10 goto 2f;	/* outer latch true branch jumps into inner loop */ \
+	goto 3f;					\
+2:	  r1 += 1;		/* inner loop */	\
+	  if r1 < 5 goto 2b;				\
+	goto 1b;					\
+3:	r0 = 0;						\
+	exit;						\
+"	::: __clobber_all);
+}
+
+/*
+ * Exact boundary hits for every continuation opcode.
+ *
+ * Example: pre_lt
+ *
+ *    u64 r6 = 1;
+ *    while (r6 < 7) {               // r6 ∈ [1,3,5,7]
+ *        r6 += 2;                   // r6 ∈ [3,5,7]
+ *    }
+ */
+/*              name               op              initial          step             bound  expected message */
+PRE__COND_TEST (pre_lt,            ">=",                 1,            2,                7, "header_count is 4 ")
+POST_COND_TEST (post_lt,           "<",                  1,            2,                7, "header_count is 3 ")
+PRE__COND_TEST (pre_le,            ">",                  1,            2,                7, "header_count is 5 ")
+POST_COND_TEST (post_le,           "<=",                 1,            2,                7, "header_count is 4 ")
+PRE__COND_TEST (pre_gt,            "<=",                 9,           -2,                3, "header_count is 4 ")
+POST_COND_TEST (post_gt,           ">",                  9,           -2,                3, "header_count is 3 ")
+PRE__COND_TEST (pre_ge,            "<",                  9,           -2,                3, "header_count is 5 ")
+POST_COND_TEST (post_ge,           ">=",                 9,           -2,                3, "header_count is 4 ")
+PRE__COND_TEST (pre_slt,           "s>=",               -5,            2,                1, "header_count is 4 ")
+POST_COND_TEST (post_slt,          "s<",                -5,            2,                1, "header_count is 3 ")
+PRE__COND_TEST (pre_sle,           "s>",                -5,            2,                1, "header_count is 5 ")
+POST_COND_TEST (post_sle,          "s<=",               -5,            2,                1, "header_count is 4 ")
+PRE__COND_TEST (pre_sgt,           "s<=",                5,           -2,               -1, "header_count is 4 ")
+POST_COND_TEST (post_sgt,          "s>",                 5,           -2,               -1, "header_count is 3 ")
+PRE__COND_TEST (pre_sge,           "s<",                 5,           -2,               -1, "header_count is 5 ")
+POST_COND_TEST (post_sge,          "s>=",                5,           -2,               -1, "header_count is 4 ")
+PRE__COND_TEST (pre_ne,            "==",                 1,            2,                7, "header_count is 4 ")
+POST_COND_TEST (post_ne,           "!=",                 1,            2,                7, "header_count is 3 ")
+PRE__COND_TEST (pre_ne_down,       "==",                 9,           -2,                3, "header_count is 4 ")
+POST_COND_TEST (post_ne_down,      "!=",                 9,           -2,                3, "header_count is 3 ")
+
+/*
+ * Bounds between two consecutive counter values.
+ *
+ * Example: pre_lt_round
+ *
+ *    u64 r6 = 1;
+ *    while (r6 < 8) {               // r6 ∈ [1,3,5,7,9]
+ *        r6 += 2;                   // r6 ∈ [3,5,7,9]
+ *    }
+ */
+PRE__COND_TEST (pre_lt_round,      ">=",                 1,            2,                8, "header_count is 5 ")
+PRE__COND_TEST (pre_le_round,      ">",                  1,            2,                8, "header_count is 5 ")
+PRE__COND_TEST (pre_gt_round,      "<=",                 9,           -2,                2, "header_count is 5 ")
+PRE__COND_TEST (pre_ge_round,      "<",                  9,           -2,                2, "header_count is 5 ")
+PRE__COND_TEST (pre_slt_round,     "s>=",               -5,            2,                2, "header_count is 5 ")
+PRE__COND_TEST (pre_sle_round,     "s>",                -5,            2,                2, "header_count is 5 ")
+PRE__COND_TEST (pre_sgt_round,     "s<=",                5,           -2,               -2, "header_count is 5 ")
+PRE__COND_TEST (pre_sge_round,     "s<",                 5,           -2,               -2, "header_count is 5 ")
+
+/*
+ * Equality on the first non-strict comparison still takes a backedge.
+ *
+ * Example: pre_le_eq
+ *
+ *    u64 r6 = 7;
+ *    while (r6 <= 7) {              // r6 ∈ [7,9]
+ *        r6 += 2;                   // r6 ∈ [9]
+ *    }
+ */
+PRE__COND_TEST (pre_le_eq,         ">",                  7,            2,                7, "header_count is 2 ")
+PRE__COND_TEST (pre_ge_eq,         "<",                  3,           -2,                3, "header_count is 2 ")
+PRE__COND_TEST (pre_sle_eq,        "s>",                 1,            2,                1, "header_count is 2 ")
+PRE__COND_TEST (pre_sge_eq,        "s<",                -1,           -2,               -1, "header_count is 2 ")
+
+/*
+ * Unsigned order across the sign bit.
+ *
+ * Example: pre_lt_sign
+ *
+ *    u64 H = 1ULL << 63, r6 = H - 2;
+ *    while (r6 < H + 1) {           // r6 ∈ [H-2,H-1,H,H+1]
+ *        r6 += 1;                   // r6 ∈ [H-1,H,H+1]
+ *    }
+ */
+PRE__COND_TEST (pre_lt_sign,       ">=",  ITER_S64_MAX - 1,            1, ITER_S64_MIN + 1, "header_count is 4 ")
+PRE__COND_TEST (pre_gt_sign,       "<=",  ITER_S64_MIN + 1,           -1, ITER_S64_MAX - 1, "header_count is 4 ")
+
+/*
+ * The first latch comparison is false.
+ *
+ * Example: pre_lt_false
+ *
+ *    u64 r6 = 7;
+ *    while (r6 < 7) {               // r6 ∈ [7]
+ *        r6 += 2;                   // unreachable
+ *    }
+ */
+PRE__COND_TEST (pre_lt_false,      ">=",                 7,            2,                7, "can't compute iterations count")
+PRE__COND_TEST (pre_le_false,      ">",                  8,            2,                7, "can't compute iterations count")
+PRE__COND_TEST (pre_gt_false,      "<=",                 3,           -2,                3, "can't compute iterations count")
+PRE__COND_TEST (pre_ge_false,      "<",                  2,           -2,                3, "can't compute iterations count")
+PRE__COND_TEST (pre_slt_false,     "s>=",                1,            2,                1, "can't compute iterations count")
+PRE__COND_TEST (pre_sle_false,     "s>",                 2,            2,                1, "can't compute iterations count")
+PRE__COND_TEST (pre_sgt_false,     "s<=",               -1,           -2,               -1, "can't compute iterations count")
+PRE__COND_TEST (pre_sge_false,     "s<",                -2,           -2,               -1, "can't compute iterations count")
+PRE__COND_TEST (pre_ne_false,      "==",                 7,            2,                7, "can't compute iterations count")
+
+/*
+ * The counter moves away from the exit; bound fallback verification.
+ *
+ * Example: pre_lt_dir
+ *
+ *    u64 r6 = 5, r9 = 0;
+ *    while (++r9 <= 8 && r6 < 7) {  // r6 ∈ [5,3,1,U64_MAX]
+ *        r6 -= 2;                   // r6 ∈ [3,1,U64_MAX]
+ *    }
+ */
+PRE__COND_TEST2(pre_lt_dir,        ">=",                 5,           -2,                7, "can't compute iterations count")
+PRE__COND_TEST2(pre_le_dir,        ">",                  5,           -2,                7, "can't compute iterations count")
+PRE__COND_TEST2(pre_gt_dir,        "<=",                 5,            2,                3, "can't compute iterations count")
+PRE__COND_TEST2(pre_ge_dir,        "<",                  5,            2,                3, "can't compute iterations count")
+PRE__COND_TEST2(pre_slt_dir,       "s>=",               -1,           -2,                1, "can't compute iterations count")
+PRE__COND_TEST2(pre_sle_dir,       "s>",                -1,           -2,                1, "can't compute iterations count")
+PRE__COND_TEST2(pre_sgt_dir,       "s<=",                1,            2,               -1, "can't compute iterations count")
+PRE__COND_TEST2(pre_sge_dir,       "s<",                 1,            2,               -1, "can't compute iterations count")
+
+/*
+ * The predicted exit would wrap the counter.
+ *
+ * Example: pre_lt_wrap
+ *
+ *    u64 M = U64_MAX, r6 = M - 1, r9 = 0;
+ *    while (++r9 <= 8 && r6 < M) {  // r6 ∈ [M-1,0,2,...,14]
+ *        r6 += 2;                   // r6 ∈ [0,2,...,14]
+ *    }
+ */
+PRE__COND_TEST2(pre_lt_wrap,       ">=",  ITER_U64_MAX - 1,            2,     ITER_U64_MAX, "can't compute iterations count")
+PRE__COND_TEST2(pre_le_wrap,       ">",       ITER_U64_MAX,            1,     ITER_U64_MAX, "can't compute iterations count")
+PRE__COND_TEST2(pre_gt_wrap,       "<=",                 1,           -2,                0, "can't compute iterations count")
+PRE__COND_TEST2(pre_ge_wrap,       "<",                  0,           -1,                0, "can't compute iterations count")
+PRE__COND_TEST2(pre_slt_wrap,      "s>=", ITER_S64_MAX - 1,            2,     ITER_S64_MAX, "can't compute iterations count")
+PRE__COND_TEST2(pre_sle_wrap,      "s>",      ITER_S64_MAX,            1,     ITER_S64_MAX, "can't compute iterations count")
+PRE__COND_TEST2(pre_sgt_wrap,      "s<=", ITER_S64_MIN + 1,           -2,     ITER_S64_MIN, "can't compute iterations count")
+PRE__COND_TEST2(pre_sge_wrap,      "s<",      ITER_S64_MIN,           -1,     ITER_S64_MIN, "can't compute iterations count")
+
+/*
+ * The non-strict count itself would overflow u64.
+ *
+ * Example: pre_le_count_ovf
+ *
+ *    u64 r6 = 0, r9 = 0;
+ *    while (++r9 <= 8 && r6 <= U64_MAX) {  // r6 ∈ [0..8]
+ *        r6 += 1;                         // r6 ∈ [1..8]
+ *    }
+ */
+PRE__COND_TEST2(pre_le_count_ovf,  ">",                  0,            1,     ITER_U64_MAX, "can't compute iterations count")
+PRE__COND_TEST2(pre_ge_count_ovf,  "<",       ITER_U64_MAX,           -1,                0, "can't compute iterations count")
+PRE__COND_TEST2(pre_sle_count_ovf, "s>",      ITER_S64_MIN,            1,     ITER_S64_MAX, "can't compute iterations count")
+PRE__COND_TEST2(pre_sge_count_ovf, "s<",      ITER_S64_MAX,           -1,     ITER_S64_MIN, "can't compute iterations count")
+
+/*
+ * JNE divisibility and equality across unsigned wraparound.
+ *
+ * Example: pre_ne_rem
+ *
+ *    u64 r6 = 1, r9 = 0;
+ *    while (++r9 <= 8 && r6 != 8) { // r6 ∈ [1,3,...,17]
+ *        r6 += 2;                   // r6 ∈ [3,5,...,17]
+ *    }
+ */
+PRE__COND_TEST2(pre_ne_rem,        "==",                 1,            2,                8, "can't compute iterations count")
+PRE__COND_TEST2(pre_ne_rem_down,   "==",                 9,           -2,                2, "can't compute iterations count")
+PRE__COND_TEST (pre_ne_wrap_up,    "==",  ITER_U64_MAX - 1,            2,                0, "header_count is 2 ")
+PRE__COND_TEST (pre_ne_wrap_down,  "==",                 1,           -2,     ITER_U64_MAX, "header_count is 2 ")
+
+/*
+ * Largest supported header count, sentinel, and u32 overflow.
+ *
+ * Example: pre_lt_max_hdrs
+ *
+ *    u64 r6 = 0, r9 = 0;
+ *    while (++r9 <= 8 && r6 < U32_MAX - 2) { // r6 ∈ [0..8]
+ *        r6 += 1;                            // r6 ∈ [1..8]
+ *    }
+ */
+PRE__COND_TEST2(pre_lt_max_hdrs,   ">=",                 0,            1, ITER_U32_MAX - 2, "header_count is [0..4294967294] ")
+PRE__COND_TEST2(pre_lt_sentinel,   ">=",                 0,            1, ITER_U32_MAX - 1, "can't compute iterations count")
+PRE__COND_TEST2(pre_lt_hdr_ovf,    ">=",                 0,            1,     ITER_U32_MAX, "can't compute iterations count")
+
+/*
+ * The extreme signed step values remain valid unsigned magnitudes.
+ *
+ * Example: pre_lt_step_max
+ *
+ *    u64 r6 = 0;
+ *    while (r6 < S64_MAX) {         // r6 ∈ [0,S64_MAX]
+ *        r6 += S64_MAX;             // r6 ∈ [S64_MAX]
+ *    }
+ */
+PRE__COND_TEST (pre_lt_step_max,   ">=",                 0, ITER_S64_MAX,     ITER_S64_MAX, "header_count is 2 ")
+PRE__COND_TEST (pre_gt_step_min,   "<=",      ITER_S64_MIN, ITER_S64_MIN,                0, "header_count is 2 ")
+PRE__COND_TEST (pre_ne_step_min,   "==",                 0, ITER_S64_MIN,     ITER_S64_MIN, "header_count is 2 ")
+
+/*
+ * A zero step cannot establish a finite iteration count.
+ *
+ * Example: pre_lt_zero_step
+ *
+ *    u64 r6 = 1, r9 = 0;
+ *    while (++r9 <= 8 && r6 < 7) {  // r6 ∈ [1], nine header visits
+ *        r6 += 0;                   // r6 ∈ [1]
+ *    }
+ */
+PRE__COND_TEST2(pre_lt_zero_step,  ">=",                 1,            0,                7, "can't compute iterations count")
+
+SEC("xdp")
+__success
+__log_level(2)
+__flag(BPF_F_TEST_STATE_FREQ)
+__msg("loop header at 1, header_count is 10 ")
+__msg("loop header at 1, widening r0 to 0..9 step 1")
+__msg("processed 5 insns")
+__naked void post_cond_jlt(void)
+{
+	asm volatile ("					\
+	r0 = 0;						\
+1:	r0 += 1;					\
+	if r0 < 10 goto 1b;				\
+	exit;						\
+"	::: __clobber_all);
+}
+
+/*
+ * Explore with r7=10 first, r7=4096 next.
+ * Should trigger invalid stack read if r7 is marked precise.
+ *
+ *   r0 = 0;
+ *   r6 = 0;
+ *   r7 = bpf_get_prandom_u32() == 42 ? 10 : 4096;
+ *   do {
+ *           r0++;
+ *           r6++;
+ *           r3 = *(fp - r0);
+ *   } while (r6 < r7);
+ *   return r0;
+ */
+SEC("xdp")
+__failure
+__log_level(2)
+__flag(BPF_F_TEST_STATE_FREQ)
+__msg("loop header at 6, header_count is 10 ")
+__msg("mark_precise: frame0: parent state regs=r6,r7 stack=:")
+__msg("loop header at 6, widening r0 to 0..9 step 1")
+__msg("loop header at 6, header_count is 4096 ")
+__msg("loop header at 6, widening r0 to 0..4095 step 1")
+__msg("invalid variable-offset read from stack R2")
+__naked void mark_bound_precise(void)
+{
+	asm volatile ("					\
+	call %[bpf_get_prandom_u32];			\
+	r7 = 4096;					\
+	if r0 != 42 goto 1f;				\
+	r7 = 10;					\
+1:	r0 = 0;						\
+	r6 = 0;						\
+2:	r0 += 1;					\
+	r6 += 1;					\
+	r2 = r0;					\
+	r2 = -r2;					\
+	r2 += r10;					\
+	r3 = *(u8 *)(r2 + 0);				\
+	if r6 < r7 goto 2b;				\
+	exit;						\
+"	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+/*
+ * Explore with r6=4086 first, r6=0 next.
+ * Should trigger invalid stack read if r6 is marked precise.
+ *
+ *   r0 = 0;
+ *   r6 = bpf_get_prandom_u32() == 42 ? 4086 : 0;
+ *   do {
+ *           r0++;
+ *           r6++;
+ *           r3 = *(fp - r0);
+ *   } while (r6 < 4096);
+ *   return r0;
+ */
+SEC("xdp")
+__failure
+__log_level(2)
+__flag(BPF_F_TEST_STATE_FREQ)
+__msg("loop header at 5, header_count is 10 ")
+__msg("mark_precise: frame0: parent state regs=r6 stack=:")
+__msg("loop header at 5, widening r0 to 0..9 step 1")
+__msg("loop header at 5, header_count is 4096 ")
+__msg("loop header at 5, widening r0 to 0..4095 step 1")
+__msg("invalid variable-offset read from stack R2")
+__naked void mark_base_precise(void)
+{
+	asm volatile ("					\
+	call %[bpf_get_prandom_u32];			\
+	r6 = 0;						\
+	if r0 != 42 goto 1f;				\
+	r6 = 4086;					\
+1:	r0 = 0;						\
+2:	r0 += 1;					\
+	r6 += 1;					\
+	r2 = r0;					\
+	r2 = -r2;					\
+	r2 += r10;					\
+	r3 = *(u8 *)(r2 + 0);				\
+	if r6 < 4096 goto 2b;				\
+	exit;						\
+"	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+/*
+ * Spill size should be taken into account when evaluating bound scev.
+ *
+ *   r0 = 256;
+ *   *(u8 *)(r10 - 8) = r0;
+ *   r0 = 0;
+ *   do {
+ *           r0++;
+ *           r1 = *(u64 *)(r10 - 8);
+ *           if (r0 & 0x200) break; // hidden from scev, limit main pass iterations.
+ *   } while (r0 < r1);
+ *   return r0;
+ */
+SEC("xdp")
+__success
+__log_level(2)
+__msg("scev at header 3:")
+__msg_next("  r0=(+ r0 1) / (linear r0 1)")
+__msg("loop header at 3, can't compute iterations count")
+__naked void eval_expr_small_spill_bound(void)
+{
+	asm volatile ("					\
+	r0 = 256;					\
+	*(u8 *)(r10 - 8) = r0;				\
+	r0 = 0;						\
+1:	r0 += 1;					\
+	r1 = *(u64 *)(r10 - 8);				\
+	if r0 & 0x200 goto 2f;				\
+	if r0 < r1 goto 1b;				\
+2:	exit;						\
+"	::: __clobber_all);
+}
+
+/* Even when the saved value fits, the upper bytes of the slot are unknown. */
+SEC("xdp")
+__success
+__log_level(2)
+__msg("scev at header 3:")
+__msg_next("  r0=(+ r0 1) / (linear r0 1)")
+__msg("loop header at 3, can't compute iterations count")
+__naked void eval_expr_small_spill_bound_fits(void)
+{
+	asm volatile ("					\
+	r0 = 10;					\
+	*(u8 *)(r10 - 8) = r0;				\
+	r0 = 0;						\
+1:	r0 += 1;					\
+	r1 = *(u64 *)(r10 - 8);				\
+	if r0 & 0x80 goto 2f;				\
+	if r0 < r1 goto 1b;				\
+2:	exit;						\
+"	::: __clobber_all);
+}
+
+/*
+ *   r0 = 255;
+ *   *(u8 *)(r10 - 8) = r0;		// narrow spill
+ *   do {
+ *           r0 = *(u64 *)(r10 - 8);	// wide load, widening should bail out here.
+ *           r0++;
+ *           *(u64 *)(r10 - 8) = r0;
+ *   } while (r0 < 265);
+ *   return r0;
+ */
+SEC("xdp")
+__success
+__log_level(2)
+__msg("scev at header 2:")
+__msg_next("  *fp-8=(+ *fp-8 1) / (linear *fp-8 1)")
+__msg("loop header at 2, can't compute iterations count")
+__not_msg("widening")
+__naked void no_widen_small_spill_counter(void)
+{
+	asm volatile ("					\
+	r0 = 255;					\
+	*(u8 *)(r10 - 8) = r0;				\
+1:	r0 = *(u64 *)(r10 - 8);				\
+	r0 += 1;					\
+	*(u64 *)(r10 - 8) = r0;				\
+	if r0 < 265 goto 1b;				\
+	exit;						\
+"	::: __clobber_all);
+}
+
+/* Full-width counterpart: the counter can be widened from 5 to 14. */
+SEC("xdp")
+__success
+__log_level(2)
+__msg("scev at header 2:")
+__msg_next("  *fp-8=(+ *fp-8 1) / (linear *fp-8 1)")
+__msg("loop header at 2, widening *fp-8 to 5..14 step 1")
+__naked void widen_full_spill_counter(void)
+{
+	asm volatile ("					\
+	r0 = 5;						\
+	*(u64 *)(r10 - 8) = r0;				\
+1:	r0 = *(u64 *)(r10 - 8);				\
+	r0 += 1;					\
+	*(u64 *)(r10 - 8) = r0;				\
+	if r0 < 15 goto 1b;				\
+	exit;						\
+"	::: __clobber_all);
+}
+
+SEC("xdp")
+__success
+__log_level(2)
+__msg("scev at header 2:")
+__msg_next("  *fp-8=(+ *fp-8 1) / (linear *fp-8 1)")
+__msg("loop header at 2, widening *fp-8 to 5..14 step 1")
+__naked void widen_acquire_release_counter(void)
+{
+	asm volatile (
+	"r0 = 5;"
+	"*(u64 *)(r10 - 8) = r0;"
+"1:"
+	".8byte %[load_acquire];"
+	"r0 += 1;"
+	".8byte %[store_release];"
+	"if r0 < 15 goto 1b;"
+	"exit;"
+	:
+	: __imm_insn(load_acquire, BPF_ATOMIC_OP(BPF_DW, BPF_LOAD_ACQ, BPF_REG_0, BPF_REG_10, -8)),
+	  __imm_insn(store_release, BPF_ATOMIC_OP(BPF_DW, BPF_STORE_REL, BPF_REG_10, BPF_REG_0, -8))
+	: __clobber_all);
+}
+
+/*
+ * r6 scev is (any r6 *fp-8). The byte store leaves memory at zero,
+ * but the spill metadata retains 256. Widen r6 to an unknown scalar
+ * instead of [1,256], which would exclude the zero loaded from memory.
+ *
+ *   r0 = 0;
+ *   *(u64 *)(r10 - 8) = r0;
+ *   r0 = 256;
+ *   *(u8 *)(r10 - 8) = r0;		// narrow spill
+ *   r6 = 1;
+ *   r7 = 0;
+ *   do {
+ *           r0 = r6;
+ *           r6 = *(u64 *)(r10 - 8);
+ *           r7++;
+ *   } while (r7 < 2);
+ *   return r0;
+ */
+SEC("xdp")
+__success __retval(0)
+__log_level(2)
+__msg("scev at header 6:")
+__msg_next("  r6=*fp-8 / (any r6 *fp-8)")
+__msg_next("  r7=(+ r7 1) / (linear r7 1)")
+__msg("loop header at 6, widening r6 to 0x8000000000000000..0x7fffffffffffffff step 1")
+__naked void widen_any_small_spill_leaf(void)
+{
+	asm volatile ("					\
+	r0 = 0;						\
+	*(u64 *)(r10 - 8) = r0;				\
+	r0 = 256;					\
+	*(u8 *)(r10 - 8) = r0;				\
+	r6 = 1;						\
+	r7 = 0;						\
+1:	r0 = r6;					\
+	r6 = *(u64 *)(r10 - 8);				\
+	r7 += 1;					\
+	if r7 < 2 goto 1b;				\
+	exit;						\
+"	::: __clobber_all);
+}
+
+/*
+ * Widening (any *fp-8 7) must allocate the slot before writing its state.
+ * The unallocated entry value makes the union an unknown scalar.
+ *
+ *   r6 = 0;
+ *   do {
+ *           if (r6 == 1)
+ *                   *(u64 *)(r10 - 8) = 7;
+ *           r0 = *(u64 *)(r10 - 8);
+ *           r6++;
+ *   } while (r6 < 3);
+ *   return r0;
+ */
+SEC("xdp")
+__success __retval(7)
+__log_level(2)
+__msg("scev at header 1:")
+__msg_next("r6=(+ r6 1) / (linear r6 1)")
+__msg_next("*fp-8=(any *fp-8 7) / (any *fp-8 7)")
+__msg("loop header at 1, widening r6 to 0..2 step 1")
+__msg("2: (7a) *(u64 *)(r10 -8) = 7          ; R10=fp0 fp-8=7 loop_stack=1")
+__naked void widen_any_unallocated_slot(void)
+{
+	asm volatile ("					\
+	r6 = 0;						\
+1:	if r6 != 1 goto 2f;				\
+	*(u64 *)(r10 - 8) = 7;				\
+2:	r0 = *(u64 *)(r10 - 8);				\
+	r6 += 1;					\
+	if r6 < 3 goto 1b;				\
+	exit;						\
+"	::: __clobber_all);
+}
+
+/* A spilled zero entry contributes zero to the union. */
+SEC("xdp")
+__success __retval(7)
+__log_level(2)
+__msg("loop header at 2, widening *fp-8 to 0..7 step 1")
+__naked void widen_any_zero_spill(void)
+{
+	asm volatile ("					\
+	*(u64 *)(r10 - 8) = 0;				\
+	r6 = 0;						\
+1:	if r6 != 1 goto 2f;				\
+	*(u64 *)(r10 - 8) = 7;				\
+2:	r0 = *(u64 *)(r10 - 8);				\
+	r6 += 1;					\
+	if r6 < 3 goto 1b;				\
+	exit;						\
+"	::: __clobber_all);
+}
+
+/* A byte store inside the slot leaves STACK_MISC and an unknown scalar value. */
+SEC("xdp")
+__success __retval(7)
+__log_level(2)
+__msg("scev at header 2:")
+__msg_next("r6=(+ r6 1) / (linear r6 1)")
+__msg_next("*fp-8=(any *fp-8 7) / (any *fp-8 7)")
+__msg("loop header at 2, widening r6 to 0..2 step 1")
+__msg("3: (79) r1 = *(u64 *)(r10 -8)         ; R1=scalar() R10=fp0 fp-8=m???????")
+__naked void widen_any_misc_slot(void)
+{
+	asm volatile ("					\
+	*(u8 *)(r10 - 1) = 1;				\
+	r6 = 0;						\
+1:	if r6 != 1 goto 2f;				\
+	r1 = *(u64 *)(r10 - 8);				\
+	*(u64 *)(r10 - 8) = 7;				\
+2:	r0 = *(u64 *)(r10 - 8);				\
+	r6 += 1;					\
+	if r6 < 3 goto 1b;				\
+	exit;						\
+"	::: __clobber_all);
+}
+
+__naked __noinline __used static int probe_read_stack(void)
+{
+	asm volatile (
+	"r2 = 8;"
+	"r3 = 0;"
+	"call %[bpf_probe_read_kernel];"
+	"exit;"
+	:
+	: __imm(bpf_probe_read_kernel)
+	: __clobber_all);
+}
+
+/* The unknown SCEV of an unbound STACK_MISC slot must not prevent widening. */
+SEC("tracepoint")
+__success
+__log_level(2)
+__msg("scev at header 4:")
+__msg_next("  r6=(+ r6 1) / (linear r6 1)")
+__msg_next("  *fp-8=?")
+__msg("loop header at 4, widening r6 to 0..9 step 1")
+__naked void widen_subprog_probe_read(void)
+{
+	asm volatile (
+	"r1 = r10;"
+	"r1 += -8;"
+	"call probe_read_stack;"
+	"r6 = 0;"
+"1:"
+	"r0 = *(u64 *)(r10 - 8);"
+	"r1 = r10;"
+	"r1 += -8;"
+	"call probe_read_stack;"
+	"r6 += 1;"
+	"if r6 < 10 goto 1b;"
+	"r0 = 0;"
+	"exit;"
+	::: __clobber_all);
+}
+
+/* An induction variable whose range overflows s64 is widened to the full range. */
+SEC("socket")
+__success
+__log_level(2)
+__msg("loop header at 4, header_count is 10")
+__msg("loop header at 4, widening r1 to 0..9 step 1")
+__msg("loop header at 4, widening r6 to 0x8000000000000000..0x7fffffffffffffff step 1")
+__flag(BPF_F_TEST_STATE_FREQ)
+__naked void widen_overflowing_range(void)
+{
+	asm volatile ("					\
+	call %[bpf_get_prandom_u32];			\
+	r6 = r0;					\
+	r6 |= 1;	/* not an unknown scalar, those need no widening */	\
+	r1 = 0;						\
+1:	r6 += 24;					\
+	r1 += 1;					\
+	if r1 < 10 goto 1b;				\
+	r0 = r6;					\
+	exit;						\
+"	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+/*
+ * The bound slot is unallocated at the first loop header. SCEV must
+ * treat it as an unknown scalar. JSET limits main-pass iterations.
+ *
+ *   r0 = 0;
+ *   do {
+ *           r0++;
+ *           r1 = *(u64 *)(r10 - 8);
+ *           if (r0 & 0x80) break;
+ *   } while (r0 < r1);
+ *   return 0;
+ */
+SEC("xdp")
+__success __retval(0)
+__log_level(2)
+__msg("scev at header 1:")
+__msg_next("  r0=(+ r0 1) / (linear r0 1)")
+__msg("loop header at 1, can't compute iterations count")
+__not_msg("widening")
+__naked void eval_expr_unallocated_stack_bound(void)
+{
+	asm volatile ("					\
+	r0 = 0;						\
+1:	r0 += 1;					\
+	r1 = *(u64 *)(r10 - 8);				\
+	if r0 & 0x80 goto 2f;				\
+	if r0 < r1 goto 1b;				\
+2:	r0 = 0;						\
+	exit;						\
+"	::: __clobber_all);
+}
+
+SEC("xdp")
+__success
+__log_level(2)
+__flag(BPF_F_TEST_STATE_FREQ)
+__msg("loop header at 3, header_count is 8")
+__msg("loop header at 3, widening r0 to 2..9 step 1")
+__msg("3: R0=scalar(smin=umin=smin32=umin32=2,smax=umax=smax32=umax32=9,{{.*}})")
+__msg("processed 6 insns")
+__naked void post_cond_jlt_with_base(void)
+{
+	asm volatile ("					\
+	r7 = 10 ll;	/* ldimm64 for a twist */	\
+	r0 = 2;						\
+1:	r0 += 1;					\
+	if r0 < r7 goto 1b;				\
+	exit;						\
+"	::: __clobber_all);
+}
+
+SEC("xdp")
+__success
+__log_level(2)
+__flag(BPF_F_TEST_STATE_FREQ)
+__msg("loop header at 1, header_count is 4 ")
+__naked void latch_base_differs_from_step(void)
+{
+	asm volatile ("					\
+	r0 = 0;						\
+1:	r1 = r0;					\
+	r1 += 1;					\
+	if r1 >= 10 goto 2f;				\
+	r0 += 4;					\
+	goto 1b;					\
+2:	r0 = r1;					\
+	exit;						\
+"	::: __clobber_all);
+}
+
+SEC("xdp")
+__success
+__log_level(2)
+__flag(BPF_F_TEST_STATE_FREQ)
+__msg("loop header at 1, header_count is 11")
+__msg("loop header at 1, widening r0 to 0..10 step 1")
+__msg("1: R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=10,{{.*}})")
+__msg("processed 5 insns")
+__naked void post_cond_jle(void)
+{
+	asm volatile ("					\
+	r0 = 0;						\
+1:	r0 += 1;					\
+	if r0 <= 10 goto 1b;				\
+	exit;						\
+"	::: __clobber_all);
+}
+
+SEC("xdp")
+__success
+__log_level(2)
+__flag(BPF_F_TEST_STATE_FREQ)
+__msg("loop header at 1, header_count is 10")
+__msg("loop header at 1, widening r0 to 0..9 step 1")
+__msg("1: R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=9,{{.*}})")
+__msg("processed 6 insns")
+__naked void post_cond_jge(void)
+{
+	asm volatile ("					\
+	r0 = 0;						\
+1:	r0 += 1;					\
+	if r0 >= 10 goto 2f;				\
+	goto 1b;					\
+2:	exit;						\
+"	::: __clobber_all);
+}
+
+SEC("xdp")
+__success
+__log_level(2)
+__flag(BPF_F_TEST_STATE_FREQ)
+__msg("loop header at 1, header_count is 11")
+__msg("loop header at 1, widening r0 to 0..10 step 1")
+__msg("1: R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=10,{{.*}})")
+__msg("processed 6 insns")
+__naked void pre_cond_jge(void)
+{
+	asm volatile ("					\
+	r0 = 0;						\
+1:	if r0 >= 10 goto 2f;				\
+	r0 += 1;					\
+	goto 1b;					\
+2:	exit;						\
+"	::: __clobber_all);
+}
+
+SEC("xdp")
+__success
+__log_level(2)
+__flag(BPF_F_TEST_STATE_FREQ)
+__msg_next("scev at header 1:")
+__msg_next("  r0=(+ r0 1) / (linear r0 1)")
+__msg_next(" scev at latch 2:")
+__msg_next("  r0=(+ r0 1) / (linear (+ r0 1) 1)")
+__msg("loop header at 1, widening r0")
+__msg("1: R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=2,{{.*}})")
+__msg("1: (07) r0 += 1                       ; R0=scalar(smin=umin=smin32=umin32=1,smax=umax=smax32=umax32=3,{{.*}})")
+__msg("2: (55) if r0 != 0x3 goto pc-2")
+__msg("3: (95) exit")
+__msg("loop header at 1, clamping r0")
+__msg("from 2 to 1: safe")
+__msg("processed 5 insns")
+__naked void post_cond_jne(void)
+{
+	asm volatile ("					\
+	r0 = 0;						\
+1:	r0 += 1;					\
+	if r0 != 3 goto 1b;				\
+	exit;						\
+"	::: __clobber_all);
+}
+
+SEC("xdp")
+__success
+__log_level(2)
+__flag(BPF_F_TEST_STATE_FREQ)
+__msg_next("scev at header 1:")
+__msg_next("  r0=(+ r0 -1) / (linear r0 -1)")
+__msg_next(" scev at latch 2:")
+__msg_next("  r0=(+ r0 -1) / (linear (+ r0 -1) -1)")
+__msg("loop header at 1, header_count is 3 ")
+__msg("loop header at 1, widening r0 to 1..3 step 1")
+__msg("1: R0=scalar(smin=umin=smin32=umin32=1,smax=umax=smax32=umax32=3,var_off=(0x0; 0x3)) loop_stack=1")
+__msg("loop header at 1, clamping r0 to 1..2 step 1")
+__msg("processed 5 insns")
+__naked void post_cond_jne_neg_step(void)
+{
+	asm volatile ("					\
+	r0 = 3;						\
+1:	r0 += -1;					\
+	if r0 != 0 goto 1b;				\
+	exit;						\
+"	::: __clobber_all);
+}
+
+SEC("xdp")
+__success
+__log_level(2)
+__flag(BPF_F_TEST_STATE_FREQ)
+__msg_next("scev at header 1:")
+__msg_next("  r0=(+ r0 1) / (linear r0 1)")
+__msg_next(" scev at latch 1:")
+__msg_next("  r0=(+ r0 1) / (linear r0 1)")
+__msg("loop header at 1, widening r0")
+__msg("1: R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=3,{{.*}})")
+__msg("1: (15) if r0 == 0x3 goto pc+2        ; R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=2,{{.*}})")
+__msg("2: (07) r0 += 1                       ; R0=scalar(smin=umin=smin32=umin32=1,smax=umax=smax32=umax32=3,{{.*}})")
+__msg("3: (05) goto pc-3")
+__msg("loop header at 1, clamping r0")
+__msg("1: safe")
+__msg("from 1 to 4: R0=3")
+__msg("4: R0=3")
+__msg("4: (95) exit")
+__msg("processed 6 insns")
+__naked void pre_cond_je1(void)
+{
+	asm volatile ("					\
+	r0 = 0;						\
+1:	if r0 == 3 goto 2f;				\
+	r0 += 1;					\
+	goto 1b;					\
+2:	exit;						\
+"	::: __clobber_all);
+}
+
+SEC("xdp")
+__success
+__log_level(2)
+__flag(BPF_F_TEST_STATE_FREQ)
+__msg("loop header at 1, header_count is [0..3] ")
+__naked void one_backedge_two_exits(void)
+{
+	asm volatile ("					\
+	r6 = 0;						\
+1:	call %[bpf_get_prandom_u32];			\
+	if r0 == 0 goto 2f;				\
+	r6 += 1;					\
+	if r6 != 3 goto 1b;				\
+2:	r0 = r6;					\
+	exit;						\
+"	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+SEC("xdp")
+__success
+__log_level(2)
+__flag(BPF_F_TEST_STATE_FREQ)
+__msg_next("scev at header 11:")
+__msg_next("  r0=(+ r0 1) / (linear r0 1)")
+__msg_next("  r1=(+ r1 2) / (linear r1 2)")
+__msg_next(" scev at latch 16:")
+__msg_next("  r0=(+ r0 1) / (linear (+ r0 1) 1)")
+__msg_next("  r1=(+ r1 2) / (linear (+ r1 2) 2)")
+__msg("loop header at 11, widening r0")
+__msg("loop header at 11, widening r1")
+__msg("11: R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=7,var_off=(0x0; 0x7)) R1=scalar(smin=smin32=0,smax=umax=smax32=umax32=14,var_off=(0x0; 0xe),step=0+2)")
+/* loop exit */
+__msg("16: (a5) if r0 < 0x8 goto pc-6")
+__msg("exiting loop 11")
+__msg("17: (95) exit")
+/* second iteration */
+__msg("loop header at 11, clamping r0")
+__msg("loop header at 11, clamping r1")
+/* iteration convergence */
+__msg("from 16 to 11: safe")
+__not_msg("{{^}}11:")
+/* map lookup error path */
+__msg("from 7 to 17: safe")
+__naked void correlated_regs(void)
+{
+	asm volatile ("					\
+	r1 = 0;						\
+	*(u64*)(r10 - 8) = r1;				\
+	r2 = r10;					\
+	r2 += -8;					\
+	r1 = %[map] ll;					\
+	call %[bpf_map_lookup_elem];			\
+	if r0 == 0 goto 2f;				\
+	r6 = r0;					\
+	r0 = 0;						\
+	r1 = 0;						\
+1:	r2 = r6;					\
+	r2 += r1;					\
+	*(u8 *)(r2 + 0) = 1;				\
+	r0 += 1;					\
+	r1 += 2;					\
+	if r0 < 8 goto 1b;				\
+2:	exit;						\
+"	:
+	: __imm(bpf_map_lookup_elem),
+	  __imm_addr(map)
+	: __clobber_all);
+}
+
+/*
+ * k = 0
+ * for (i = 0; i < 4; i++):
+ *   for (j = 0; j < 4; j++):
+ *     k += 1
+ *     k <<= 1   // make SCEV construction not possible
+ *     k >>= 1
+ * map[k] = 1    // make k precise
+ */
+SEC("xdp")
+__success
+__log_level(2)
+__flag(BPF_F_TEST_STATE_FREQ)
+__msg("19: (72) *(u8 *)(r4 +0) = 1           ; R4=map_value(id={{.*}},map=map,ks=4,vs=1024,imm=16)")
+__not_msg("19: ")
+__msg("processed 106 insns")
+__naked void nested_loops_precise_var1(void)
+{
+	asm volatile ("					\
+	*(u64*)(r10 - 8) = 0;				\
+	r1 = %[map] ll;					\
+	r2 = r10;					\
+	r2 += -8;					\
+	call %[bpf_map_lookup_elem];			\
+	if r0 == 0 goto 3f;				\
+	r1 = 0;						\
+	r3 = 0;						\
+	/* outer loop */				\
+1:	r2 = 0;						\
+	/* inner loop */				\
+2:	r2 += 1;					\
+	r3 += 1;					\
+	r3 <<= 1;					\
+	r3 >>= 1;					\
+	if r2 < 4 goto 2b;				\
+	r1 += 1;					\
+	if r1 < 4 goto 1b;				\
+	r4 = r0;					\
+	r4 += r3;					\
+	*(u8 *)(r4 + 0) = 1;				\
+	r0 = 0;						\
+3:	exit;						\
+"	:
+	: __imm(bpf_map_lookup_elem),
+	  __imm_addr(map)
+	: __clobber_all);
+}
+
+SEC("xdp")
+__success
+__log_level(2)
+__msg("loop header at 2, header_count is 100 ")
+__msg("loop header at 4, header_count is [0..100] ")
+__msg("loop header at 6, header_count is [0..100] ")
+__msg("processed 20 insns")
+__flag(BPF_F_TEST_STATE_FREQ)
+__naked void nested_loop_with_two_exits(void)
+{
+	asm volatile ("					\
+	call %[bpf_get_prandom_u32];			\
+	r6 = 0;						\
+1:	r6 += 1;					\
+	r7 = 0;						\
+2:	r7 += 1;					\
+	r8 = 0;						\
+3:	r8 += 1;					\
+	call %[bpf_get_prandom_u32];			\
+	if r0 == 42 goto +1;				\
+	goto 4f;					\
+	if r8 < 100 goto 3b;				\
+	if r7 < 100 goto 2b;				\
+4:	if r6 < 100 goto 1b;				\
+	exit;						\
+"	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+SEC("xdp")
+__success
+__log_level(2)
+__naked void exit_loop_into_loop_header(void)
+{
+	asm volatile ("					\
+	r1 = 0;						\
+	r2 = 0;						\
+loop_a_%=:						\
+	r1 += 1;					\
+	if r1 < 10 goto loop_a_%=;			\
+loop_b_%=:						\
+	r2 += 1;					\
+	if r2 < 10 goto loop_b_%=;			\
+	r0 = 0;						\
+	exit;						\
+"	::: __clobber_all);
+}
+
+/*
+ * This exercises verifier.c:loop_stack_{pop,push}() implementation,
+ * at 'goto d' loops 'b' and 'a' have to be popped from stack,
+ * while loops 'c' and 'd' have to be pushed to stack.
+ *
+ *   loop a:                  // header 5
+ *     loop b:                // header 6
+ *       if (rand) goto d;    // 8 -> 12, side entry into inner loop d
+ *       ...
+ *   loop c:                  // header 11
+ *     loop d:                // header 12
+ *       ...
+ */
+SEC("xdp")
+__log_level(2)
+__msg("loop at 5")
+__msg("loop at 6, nested in 5")
+__msg("loop at 11, irreducible")
+__msg("loop at 12, nested in 11")
+/* entry via if r0 == 5 goto d_%= false branch */
+__msg("loop header at 12, header_count is 3 ")
+__msg("loop header at 12, widening r9 to 0..2 step 1")
+__msg("12: R8=1 R9=scalar(smin=smin32=0,smax=umax=smax32=umax32=2,var_off=(0x0; 0x3)) loop_stack=11,12")
+/* entry via if r0 == 5 goto d_%= true branch */
+__msg("loop header at 12, header_count is 3 ")
+__msg("loop header at 12, widening r9 to 0..2 step 1")
+__msg("from 8 to 12: R8=0 R9=scalar(smin=smin32=0,smax=umax=smax32=umax32=2,var_off=(0x0; 0x3)) R10=fp0 loop_stack=11,12")
+__naked void enter_nested_loop_from_side(void)
+{
+	asm volatile ("					\
+	call %[bpf_get_prandom_u32];			\
+	r6 = 0;						\
+	r7 = 0;						\
+	r8 = 0;						\
+	r9 = 0;						\
+a_%=:	r6 += 1;					\
+b_%=:	r7 += 1;					\
+	call %[bpf_get_prandom_u32];			\
+	if r0 == 5 goto d_%=;				\
+	if r7 < 3 goto b_%=;				\
+	if r6 < 3 goto a_%=;				\
+c_%=:	r8 += 1;					\
+d_%=:	r9 += 1;					\
+	if r9 < 3 goto d_%=;				\
+	if r8 < 3 goto c_%=;				\
+	r0 = 0;						\
+	exit;						\
+"	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+/*
+ * Loop A (counter in r6) contains irreducible loop B (header 5),
+ * which contains C (counter in r9).
+ * Entering B at 'body' saves and restores r6, making it appear invariant.
+ * Entering B at 'alternate' skips the save and modifies r6 instead.
+ * Hence A must not infer a SCEV expression for r6.
+ * SCEV expression for r9 in C should still be computed.
+ *
+ *  0: r6 = 0;
+ *     do {                              // A
+ *  1:     r6++;
+ *  2:     r0 = bpf_get_prandom_u32();
+ *  3:     r7 = 0;
+ *  4:     if (r0 > 5) goto alternate;
+ *  5: B:  r8 = r6;                      // B
+ *  6:     goto body;
+ *  7: alternate:
+ *         r8 = r6;
+ *  8:     r8++;
+ *  9: body:
+ *         r6 = r8;
+ * 10:     r9 = 0;
+ *         do {                          // C
+ * 11:         r9++;
+ * 12:     } while (r9 < 3);
+ * 13:     r7++;
+ * 14:     if (r7 < 4) goto B;
+ * 15: } while (r6 < 4);
+ * 16: r0 = 0;
+ * 17: return r0;
+ */
+SEC("xdp")
+__success
+__log_level(2)
+__msg("loop at 1{{$}}")
+__msg("loop at 5, nested in 1, irreducible")
+__msg("loop at 11, nested in 5")
+__msg("scev at header 1:")
+__msg_next("  r6=?")
+__msg("scev at header 11:")
+__msg_next("  r9=(+ r9 1) / (linear r9 1)")
+__msg("loop header at 11, widening r9 to 0..2 step 1")
+__not_msg("loop header at 1, widening r6")
+__naked void nested_irreducible_loop(void)
+{
+	asm volatile ("					\
+	r6 = 0;						\
+a_%=:	r6 += 1;					\
+	call %[bpf_get_prandom_u32];			\
+	r7 = 0;						\
+	if r0 > 5 goto alternate_%=;			\
+b_%=:	r8 = r6;					\
+	goto body_%=;					\
+alternate_%=:						\
+	r8 = r6;					\
+	r8 += 1;					\
+body_%=:						\
+	r6 = r8;					\
+	r9 = 0;						\
+c_%=:	r9 += 1;					\
+	if r9 < 3 goto c_%=;				\
+	r7 += 1;					\
+	if r7 < 4 goto b_%=;				\
+	if r6 < 4 goto a_%=;				\
+	r0 = 0;						\
+	exit;						\
+"	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+/*
+ * Loop A (counter in r6) contains irreducible loop B.
+ * Loop B has an exit edge from A. This is a corner case
+ * observed during testing: insn_aux_data[l1].need_scev == true,
+ * but SCEV environment is not constructed.
+ *
+ *	  r6 = 0;
+ *	  do {					// A
+ *	     r6++;
+ *	     r0 = bpf_get_prandom_u32();
+ *	     r7 = 0;
+ *	     if (r0 > 5) goto alternate;
+ *	     do {				// B
+ *	       r7++;
+ * alt:	       r0 = bpf_get_prandom_u32();
+ * l1:	       if (r0 == 42)
+ *		 goto exit;
+ *	       r7++;
+ *	     } while (r7 < 4);
+ *	  } while (r6 < 4);
+ *	  r0 = 0;
+ * exit:  return r0;
+ */
+SEC("xdp")
+__success
+__log_level(2)
+__naked void nested_irreducible_loop2(void)
+{
+	asm volatile ("					\
+	r6 = 0;						\
+a_%=:	r6 += 1;					\
+	call %[bpf_get_prandom_u32];			\
+	r7 = 0;						\
+	if r0 > 5 goto alternate_%=;			\
+b_%=:	r7 += 1;					\
+alternate_%=:						\
+	call %[bpf_get_prandom_u32];			\
+	if r0 == 42 goto exit_%=;			\
+	r7 += 1;					\
+	if r7 < 4 goto b_%=;				\
+	if r6 < 4 goto a_%=;				\
+	r0 = 0;						\
+exit_%=:exit;						\
+"	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+/*
+ * Induction variable seeded from a non-constant value. r7 enters the loop as a
+ * range aligned to 2 (prandom & 0x6 -> {0,2,4,6}) and is incremented by a
+ * non-power-of-2 slope of 6. Since the entry value is not a single point, only
+ * the power-of-two alignment shared by the entry value and the slope can be
+ * guaranteed, so the widened step is 2.
+ */
+SEC("xdp")
+__success
+__log_level(2)
+__flag(BPF_F_TEST_STATE_FREQ)
+__msg("loop header at 4, widening r7 to 0..18 step 2")
+__msg("R7=scalar(smin=smin32=0,smax=umax=smax32=umax32=18,var_off=(0x0; 0x1e),step=0+2)")
+__naked void widen_nonconst_base(void)
+{
+	asm volatile ("					\
+	call %[bpf_get_prandom_u32];			\
+	r7 = r0;					\
+	r7 &= 0x6;					\
+	r6 = 0;						\
+1:	r7 += 6;					\
+	r6 += 1;					\
+	if r6 < 3 goto 1b;				\
+	r0 = 0;						\
+	exit;						\
+"	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+/*
+ * After excluding zero from {0,4,8,12}, the interval starts at 1 while the
+ * values remain multiples of 4. Widening over three iterations must preserve
+ * base 0 and include 20.
+ */
+SEC("xdp")
+__success
+__log_level(2)
+__flag(BPF_F_TEST_STATE_FREQ)
+__msg("loop header at 5, header_count is 3 ")
+__msg("5: R7=scalar({{.*}}smax=umax=smax32=umax32=20,{{.*}}step=0+4)")
+__naked void widen_nonconst_base_refined(void)
+{
+	asm volatile ("					\
+	call %[bpf_get_prandom_u32];			\
+	r7 = r0;					\
+	r7 &= 0xc;					\
+	if r7 < 1 goto 2f;				\
+	r8 = 0;						\
+1:	r7 += 4;					\
+	r8 += 1;					\
+	if r8 < 3 goto 1b;				\
+2:	r0 = 0;						\
+	exit;						\
+"	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+/*
+ * Two nested loops with (1) exiting directly to (2):
+ *
+ *   for (r6 = 0; r6 < 4; r6++) {
+ *     r7 = 0;
+ *     for (; r7 <  3; r7++) {}   // (1)
+ *     for (; r7 != 0; r7--) {}   // (2)
+ *   }
+ */
+SEC("xdp")
+__success
+__log_level(2)
+__msg("loop header at 1, widening r6 to 0..3 step 1")
+__msg("loop header at 2, widening r7 to 0..2 step 1")
+__msg("exiting loop 2")
+__msg("entering loop 4")
+__msg("loop header at 4, widening r7 to 1..3 step 1")
+__naked void sibling_inner_loops(void)
+{
+	asm volatile ("					\
+	r6 = 0;						\
+1:	r7 = 0;						\
+2:	r7 += 1;					\
+	if r7 < 3 goto 2b;				\
+3:	r7 += -1;					\
+	if r7 != 0 goto 3b;				\
+	r6 += 1;					\
+	if r6 < 4 goto 1b;				\
+	r0 = 0;						\
+	exit;						\
+"	::: __clobber_all);
+}
+
+SEC("socket")
+__success
+__log_level(2)
+__msg("loop header at 0, can't compute iterations count")
+__naked void uninit_slot_counter(void)
+{
+	asm volatile ("					\
+1:	r0 = *(u64 *)(r10 - 8);				\
+	r0 += 1;					\
+	*(u64 *)(r10 - 8) = r0;				\
+	if r0 < 10 goto 1b;				\
+	r0 = 0;						\
+	exit;						\
+"	::: __clobber_all);
+}
+
+/*
+ * The assignment happens only on the second iteration:
+ *
+ *   r7 = 5;
+ *   for (r6 = 0; r6 < 3; r6++)
+ *           if (r6 == 1)
+ *                   r7 = 10;
+ *   if (r7 != 10)
+ *           invalid_stack_read();
+ *
+ * R7 is always 10 at the real exit. Widening loses the correlation between
+ * R6 and R7, leaving R7 in [5, 10] both at the loop header and after the loop.
+ * The verifier therefore rejects the possible invalid stack read.
+ */
+SEC("xdp")
+__failure
+__log_level(2)
+__msg("2: {{.*}}R7=scalar(smin=umin=smin32=umin32=5,smax=umax=smax32=umax32=10,var_off=(0x0; 0xf))")
+__msg("7: R7=scalar(smin=umin=smin32=umin32=5,smax=umax=smax32=umax32=10,var_off=(0x0; 0xf))")
+__msg("invalid read from stack R10 off=0 size=8")
+__naked void conditional_assignment_on_second_iteration(void)
+{
+	asm volatile ("					\
+	r7 = 5;						\
+	r6 = 0;						\
+1:	if r6 >= 3 goto 3f;				\
+	if r6 != 1 goto 2f;				\
+	r7 = 10;					\
+2:	r6 += 1;					\
+	goto 1b;					\
+3:	if r7 == 10 goto 4f;				\
+	r0 = *(u64 *)(r10 + 0);				\
+4:	r0 = 0;						\
+	exit;						\
+"	::: __clobber_all);
+}
+
+/*
+ * The latch bound has header SCEV (any r7 100), so it is not invariant:
+ *
+ *   u64 i = 0, n = 3;
+ *   for (;;) {
+ *           ++i;
+ *           if (i >= n)
+ *                   break;
+ *           if (i == 1)
+ *                   n = 100;
+ *   }
+ *
+ * The ANY expression must also appear at the latch, rather than a bare r7
+ * that would incorrectly be treated as a loop-invariant bound.
+ */
+SEC("xdp")
+__success
+__log_level(2)
+__msg("scev at header 2:")
+__msg_next("  r6=(+ r6 1) / (linear r6 1)")
+__msg_next("  r7=(any r7 100) / (any r7 100)")
+__msg_next(" scev at latch 3:")
+__msg_next("  r6=(+ r6 1) / (linear (+ r6 1) 1)")
+__msg_next("  r7=r7 / (any r7 100)")
+__naked void no_widen_changing_latch_bound(void)
+{
+	asm volatile ("					\
+	r6 = 0;						\
+	r7 = 3;						\
+loop_%=:						\
+	r6 += 1;					\
+	if r6 >= r7 goto exit_%=;			\
+	if r6 != 1 goto next_%=;			\
+	r7 = 100;					\
+next_%=:						\
+	goto loop_%=;					\
+exit_%=:						\
+	r0 = 0;						\
+	exit;						\
+"	::: __clobber_all);
+}
+
+/*
+ * Join two stack pointer offsets via (any r7 r8), with r8 loop-invariant.
+ * The joined range [-16, -8] still points to initialized stack memory,
+ * so dereferencing r7 after the loop is safe.
+ */
+SEC("xdp")
+__success __retval(0)
+__log_level(2)
+__msg("loop header at 7, widening r7 to -16..-8 step 1")
+__msg("7: {{.*}}R7=fp(smin=smin32=-16,smax=smax32=-8,")
+__msg("12: R7=fp(smin=smin32=-16,smax=smax32=-8,")
+__naked void conditional_stack_pointer_assignment(void)
+{
+	asm volatile ("					\
+	*(u64 *)(r10 - 16) = 0;				\
+	*(u64 *)(r10 - 8) = 0;				\
+	r7 = r10;					\
+	r7 += -16;					\
+	r8 = r10;					\
+	r8 += -8;					\
+	r6 = 0;						\
+1:	if r6 >= 3 goto 3f;				\
+	if r6 != 1 goto 2f;				\
+	r7 = r8;					\
+2:	r6 += 1;					\
+	goto 1b;					\
+3:	r0 = *(u64 *)(r7 + 0);				\
+	exit;						\
+"	::: __clobber_all);
+}
+
+/*
+ * Both choices are valid, but combining their offsets while retaining
+ * small_map's metadata would invent small_map + 32, outside its value.
+ *
+ *   r6 = lookup(small_map, 0);
+ *   r7 = lookup(map, 0) + 32;
+ *   for (r8 = 0; r8 < 3; r8++) {
+ *           if (r8 == 1)
+ *                   r6 = r7;
+ *           r0 = *(u32 *)(r6 + 0);
+ *   }
+ */
+SEC("xdp")
+__success __retval(0)
+__log_level(2)
+__msg("loop header at {{[0-9]+}}, can't widen r6, expr is")
+__not_msg("widening")
+__naked void no_widen_any_map_origin(void)
+{
+	asm volatile ("					\
+	*(u64 *)(r10 - 8) = 0;				\
+	r2 = r10;					\
+	r2 += -8;					\
+	r1 = %[small_map] ll;				\
+	call %[bpf_map_lookup_elem];			\
+	if r0 == 0 goto 3f;				\
+	r6 = r0;					\
+	r2 = r10;					\
+	r2 += -8;					\
+	r1 = %[map] ll;					\
+	call %[bpf_map_lookup_elem];			\
+	if r0 == 0 goto 3f;				\
+	r7 = r0;					\
+	r7 += 32;					\
+	r8 = 0;						\
+1:	if r8 != 1 goto 2f;				\
+	r6 = r7;					\
+2:	r0 = *(u32 *)(r6 + 0);				\
+	r8 += 1;					\
+	if r8 < 3 goto 1b;				\
+3:	r0 = 0;						\
+	exit;						\
+"	:
+	: __imm(bpf_map_lookup_elem), __imm_addr(small_map), __imm_addr(map)
+	: __clobber_all);
+}
+
+__naked __noinline __used static int any_stack_origin_callee(void)
+{
+	asm volatile ("					\
+	*(u64 *)(r10 - 8) = 0;				\
+	r6 = r1;					\
+	r7 = r10;					\
+	r7 += -8;					\
+	r8 = 0;						\
+1:	if r8 != 1 goto 2f;				\
+	r6 = r7;					\
+2:	r8 += 1;					\
+	if r8 < 3 goto 1b;				\
+	r0 = *(u64 *)(r6 + 0);				\
+	exit;						\
+"	::: __clobber_all);
+}
+
+/*
+ * The callee chooses between caller fp-16 and its own fp-8:
+ *
+ *   r6 = r1;
+ *   r7 = fp - 8;
+ *   for (r8 = 0; r8 < 3; r8++)
+ *           if (r8 == 1)
+ *                   r6 = r7;
+ *   return *(u64 *)r6;
+ *
+ * Both locations hold zero. Their offsets belong to different frames
+ * and cannot be combined into one stack pointer.
+ */
+SEC("xdp")
+__success __retval(0)
+__log_level(2)
+__msg("loop header at {{[0-9]+}}, can't widen r6, expr is")
+__not_msg("widening")
+__naked void no_widen_any_stack_origin(void)
+{
+	asm volatile ("					\
+	*(u64 *)(r10 - 16) = 0;				\
+	r1 = r10;					\
+	r1 += -16;					\
+	call any_stack_origin_callee;			\
+	exit;						\
+"	::: __clobber_all);
+}
+
+/*
+ * r7 starts as 1 + 6*k and invariant r8 as 1 + 9*k (0 <= k <= 3).
+ * The (any r7 r8) union must preserve base 1 and step gcd(6, 9) = 3.
+ */
+SEC("xdp")
+__success __retval(1)
+__log_level(2)
+__msg("r7 += 1 {{.*}}step=1+6)")
+__msg("r8 += 1 {{.*}}step=1+9)")
+__msg("loop header at 9, widening r7 to 1..28 step 3")
+__msg("9: {{.*}}R7=scalar({{.*}},step=1+3)")
+__msg("14: R7=scalar({{.*}},step=1+3)")
+__naked void conditional_scalar_assignment_gcd(void)
+{
+	asm volatile ("					\
+	call %[bpf_get_prandom_u32];			\
+	r0 &= 3;					\
+	r7 = r0;					\
+	r7 *= 6;					\
+	r7 += 1;					\
+	r8 = r0;					\
+	r8 *= 9;					\
+	r8 += 1;					\
+	r6 = 0;						\
+1:	if r6 >= 3 goto 3f;				\
+	if r6 != 1 goto 2f;				\
+	r7 = r8;					\
+2:	r6 += 1;					\
+	goto 1b;					\
+3:	r0 = r7;					\
+	r0 %%= 3;					\
+	exit;						\
+"	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+/*
+ * r7 = ctx->data;
+ * r8 = r7
+ * for (r6 = 0; r6 < 10 && random() != 42; r7++, r6++);
+ * if (r7 >= ctx->data_end)
+ *   return;
+ * *(r8 + 4);  // At this point the loop executed unknown number of times.
+ *	       // Hence, r7 range gives no information about r8.
+ */
+SEC("tc")
+__failure
+__msg("R8 min value is outside of the allowed memory range")
+__naked void break_pkt_pointers_id(void)
+{
+	asm volatile ("					\
+	r7 = *(u32*)(r1 + %[__sk_buff_data]);		\
+	r8 = r7;					\
+	r9 = *(u32*)(r1 + %[__sk_buff_data_end]);	\
+	r6 = 0;						\
+1:	call %[bpf_get_prandom_u32];			\
+	if r0 == 42 goto 3f;				\
+	r6 += 1;					\
+	r7 += 1;					\
+	if r6 < 10 goto 1b;				\
+3:	if r7 >= r9 goto 2f;				\
+	r0 = *(u8*)(r8 + 4);				\
+2:	r0 = 0;						\
+	exit;						\
+"	:
+	: __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)),
+	  __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)),
+	  __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+/*
+ * A loop induction variable used to compute the base address of a store to the
+ * stack must not be widened: the spill offset would become varying, which the
+ * verifier does not track.
+ */
+SEC("xdp")
+__success
+__log_level(2)
+__flag(BPF_F_TEST_STATE_FREQ)
+__msg("loop header at 2, can't widen r2, expr is (+ r2 8), requires exact stack-offset tracking")
+__naked void no_widen_stack_spill(void)
+{
+	asm volatile ("					\
+	r0 = 0;						\
+	r2 = 0;						\
+1:	r3 = r10;					\
+	r3 += -64;					\
+	r3 += r2;					\
+	*(u64 *)(r3 + 0) = r0;				\
+	r0 += 1;					\
+	r2 += 8;					\
+	if r0 < 4 goto 1b;				\
+	r0 = 0;						\
+	exit;						\
+"	::: __clobber_all);
+}
+/*
+ * Same hazard across a loop nest: the outer induction variable r2 addresses a
+ * stack store performed inside the inner loop. The dependency is pulled up from
+ * the inner loop, so the outer loop must not widen r2.
+ */
+SEC("xdp")
+__success
+__log_level(2)
+__flag(BPF_F_TEST_STATE_FREQ)
+__msg("loop header at 2, can't widen r2, expr is (+ r2 8), requires exact stack-offset tracking")
+__msg("loop header at 3, widening r1 to 0..1 step 1")
+__naked void no_widen_stack_spill_nested(void)
+{
+	asm volatile ("					\
+	r0 = 0;						\
+	r2 = 0;						\
+1:	r1 = 0;						\
+2:	r3 = r10;					\
+	r3 += -64;					\
+	r3 += r2;					\
+	*(u64 *)(r3 + 0) = r1;				\
+	r1 += 1;					\
+	if r1 < 2 goto 2b;				\
+	r0 += 1;					\
+	r2 += 8;					\
+	if r0 < 4 goto 1b;				\
+	r0 = 0;						\
+	exit;						\
+"	::: __clobber_all);
+}
+
+/*
+ * The inner loop restores r6 on its backedge but exits with r6 = -1000 on break.
+ * Outer loop must account for the exit value instead of deriving r6 = r6_entry + n.
+ * On the first outer backedge r6 is -999, so the next map access is invalid.
+ *
+ *   u8 *r7 = map_value;
+ *   s64 r6 = 0;
+ *   do {
+ *   1:       r9 = 0;
+ *           r0 = r7[r6];
+ *           while (true) {
+ *   2:              r8 = r6;
+ *                   r6 = -1000;
+ *                   if (++r9 > 5)
+ *                           break;
+ *                   r6 = r8;
+ *   3:      }
+ *           r6++;
+ *   } while (r6 < 3);
+ */
+SEC("xdp")
+__failure
+__log_level(2)
+__flag(BPF_F_TEST_STATE_FREQ)
+__msg("scev at header 9:")
+__msg_next("  r6=(+ ? 1) / ?")
+__msg_next(" scev at latch 20:")
+__msg_next("  r6=(+ ? 1) / (+ ? 1)")
+__msg_next("scev at header 13:")
+__msg_next("  r9=(+ r9 1) / (linear r9 1)")
+__msg_next(" scev at latch 16:")
+__msg_next("  r6=-1000 / -1000")
+__msg_next("  r8=r6 / r6")
+__msg_next("  r9=(+ r9 1) / (linear (+ r9 1) 1)")
+__msg("R6=-999")
+__msg("R1 min value is negative")
+__naked void nested_loop_exit_clobbers_reg(void)
+{
+	asm volatile (
+	"*(u64 *)(r10 - 8) = 0;"
+	"r2 = r10;"
+	"r2 += -8;"
+	"r1 = %[map] ll;"
+	"call %[bpf_map_lookup_elem];"
+	"if r0 == 0 goto 4f;"
+	"r7 = r0;"
+	"r6 = 0;"
+"1:"
+	"r9 = 0;"
+	"r1 = r7;"
+	"r1 += r6;"
+	"r0 = *(u8 *)(r1 + 0);"
+"2:"
+	"r8 = r6;"
+	"r6 = -1000;"
+	"r9 += 1;"
+	"if r9 > 5 goto 3f;"
+	"r6 = r8;"
+	"goto 2b;"
+"3:"
+	"r6 += 1;"
+	"if r6 s< 3 goto 1b;"
+"4:"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_map_lookup_elem),
+	  __imm_addr(map)
+	: __clobber_all);
+}
+
+/*
+ * The inner loop exits both nested loops with r6 unchanged, or exits only itself with r6 = -1000.
+ * The middle loop repairs the latter exit with a stack fill. It therefore preserves r6,
+ * so the outermost loop must retain r6_entry + n.
+ *
+ *   r6 = 0;
+ *   do {
+ *           r7 = 0;
+ *           do {
+ *                   spill = r6;
+ *                   r9 = 0;
+ *                   do {
+ *                           r8 = r6;
+ *                           if (random() & 1)
+ *                                   goto next;
+ *                           r6 = -1000;
+ *                           if (++r9 >= 2)
+ *                                   break;
+ *                           r6 = r8;
+ *                   } while (true);
+ *                   r6 = spill;
+ *           } while (++r7 < 2);
+ *   next:
+ *           r6++;
+ *   } while (r6 < 3);
+ */
+SEC("xdp")
+__success __retval(3)
+__log_level(2)
+__flag(BPF_F_TEST_STATE_FREQ)
+__msg("r6 = *(u64 *)(r10 -8) {{.*}}; r6 ? -> r6")
+__msg("scev at header 1:")
+__msg_next("  r6=(+ r6 1) / (linear r6 1)")
+__msg_next(" scev at latch 16:")
+__msg_next("  r6=(+ r6 1) / (linear (+ r6 1) 1)")
+__msg("scev at header 2:")
+__msg_next("  r7=(+ r7 1) / (linear r7 1)")
+__msg("scev at header 4:")
+__msg_next("  r9=(+ r9 1) / (linear r9 1)")
+__msg(" scev at latch 9:")
+__msg_next("  r8=r6 / r6")
+__msg_next("  r9=(+ r9 1) / (linear (+ r9 1) 1)")
+__naked void nested_loop_exit_preserves_reg(void)
+{
+	asm volatile (
+	"r6 = 0;"
+"1:"
+	"r7 = 0;"
+"2:"
+	"*(u64 *)(r10 - 8) = r6;"
+	"r9 = 0;"
+"3:"
+	"r8 = r6;"
+	"call %[bpf_get_prandom_u32];"
+	"if r0 & 1 goto 5f;"
+	"r6 = -1000;"
+	"r9 += 1;"
+	"if r9 >= 2 goto 4f;"
+	"r6 = r8;"
+	"goto 3b;"
+"4:"
+	"r6 = *(u64 *)(r10 - 8);"
+	"r7 += 1;"
+	"if r7 < 2 goto 2b;"
+"5:"
+	"r6 += 1;"
+	"if r6 < 3 goto 1b;"
+	"r0 = r6;"
+	"exit;"
+	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+/*
+ * Complement of no_widen_stack_spill: a sub-register (1-byte) store to the stack
+ * lands as STACK_MISC and carries no tracked value, so the induction variable
+ * addressing it (r2) is still widened.
+ */
+SEC("xdp")
+__success
+__log_level(2)
+__flag(BPF_F_TEST_STATE_FREQ)
+__msg("widening r2 to 0..24 step 8")
+__naked void widen_byte_stack_store(void)
+{
+	asm volatile ("					\
+	r0 = 0;						\
+	r2 = 0;						\
+1:	r3 = r10;					\
+	r3 += -64;					\
+	r3 += r2;					\
+	*(u8 *)(r3 + 0) = r0;				\
+	r0 += 1;					\
+	r2 += 8;					\
+	if r0 < 4 goto 1b;				\
+	r0 = 0;						\
+	exit;						\
+"	::: __clobber_all);
+}
+
+/*
+ * A fill (BPF_LDX) at a varying stack offset loses precision just like a spill,
+ * so the induction variable computing the load base (r2) must not be widened.
+ * The slots are initialized up front so the fill itself is a valid read.
+ */
+SEC("xdp")
+__success
+__log_level(2)
+__flag(BPF_F_TEST_STATE_FREQ)
+__msg("loop header at 6, can't widen r2, expr is (+ r2 8), requires exact stack-offset tracking")
+__naked void no_widen_stack_fill(void)
+{
+	asm volatile ("					\
+	r0 = 0;						\
+	*(u64 *)(r10 - 64) = r0;			\
+	*(u64 *)(r10 - 56) = r0;			\
+	*(u64 *)(r10 - 48) = r0;			\
+	*(u64 *)(r10 - 40) = r0;			\
+	r2 = 0;						\
+1:	r3 = r10;					\
+	r3 += -64;					\
+	r3 += r2;					\
+	r4 = *(u64 *)(r3 + 0);				\
+	r0 += 1;					\
+	r2 += 8;					\
+	if r0 < 4 goto 1b;				\
+	r0 = 0;						\
+	exit;						\
+"	::: __clobber_all);
+}
+
+/*
+ * struct xdp_md *slots[2] = { ctx, ctx };
+ *
+ * r7 = 0;
+ * for (r8 = 0; r8 < 2; r8++) {
+ *         value = slots[r7]->data;
+ *         if (i == 0)
+ *                 r7 = 1;
+ * }
+ *
+ * SCEV expression for r7 is (any r7 8) and r7 is used to
+ * address the stack memory. Avoid widening it, otherwise
+ * verifier won't know the type of the slots[r7] expression.
+ */
+SEC("xdp")
+__success
+__log_level(2)
+__flag(BPF_F_TEST_STATE_FREQ)
+__msg("scev at header 4:")
+__msg_next("r7=(any r7 8) / (any r7 8)")
+__msg("loop header at 4, can't widen r7, expr is {{.*}}, requires exact stack-offset tracking")
+__naked void no_widen_stack_fill_any(void)
+{
+	asm volatile ("					\
+	*(u64 *)(r10 - 16) = r1;			\
+	*(u64 *)(r10 - 8) = r1;				\
+	r7 = 0;						\
+	r8 = 0;						\
+1:	r2 = r10;					\
+	r2 += -16;					\
+	r2 += r7;					\
+	r3 = *(u64 *)(r2 + 0);	/* slots[r7] */		\
+	r0 = *(u32 *)(r3 + 0);	/* slots[r7]->data */	\
+	if r8 != 0 goto 2f;				\
+	r7 = 8;			/* conditionally update r7 */ \
+2:	r8 += 1;					\
+	if r8 < 2 goto 1b;				\
+	r0 = 0;						\
+	exit;						\
+"	::: __clobber_all);
+}
+
+/*
+ * A dynptr/iter/irq/res_spin_lock call initializes a stack object through a
+ * pointer argument, which acts like a spill base: the induction variable
+ * computing that argument's varying stack offset must not be widened, otherwise
+ * the slot can't be resolved. Here each iteration constructs an xdp dynptr at
+ * &dptrs[i].
+ */
+SEC("xdp")
+__success
+__log_level(2)
+__flag(BPF_F_TEST_STATE_FREQ)
+__msg("can't widen {{.*}}, requires exact stack-offset tracking")
+#ifndef __clang__
+/* The issue is unrelated to SCEV */
+__skip("GCC emits a stack-pointer loop condition the verifier cannot resolve")
+#endif
+int no_widen_dynptr_kfunc_arg(struct xdp_md *ctx)
+{
+	struct bpf_dynptr dptrs[4];
+	int i;
+
+#ifdef __clang__
+#pragma clang loop unroll(disable)
+#else
+#pragma GCC unroll 0
+#endif
+	for (i = 0; i < 4; i++)
+		bpf_dynptr_from_xdp(ctx, 0, &dptrs[i]);
+
+	return 0;
+}
+
+SEC("xdp")
+__success
+__log_level(2)
+__flag(BPF_F_TEST_STATE_FREQ)
+__msg("header_count is 4 ")
+__msg("widening r6 to -16..-13 step 1")
+__naked void ptr_stack(void)
+{
+	asm volatile ("					\
+	r6 = r10;					\
+	r6 += -16;					\
+	r7 = r6;					\
+	r7 += 4;					\
+1:	r6 += 1;					\
+	if r6 < r7 goto 1b;				\
+	r0 = 0;						\
+	exit;						\
+"	::: __clobber_all);
+}
+
+__naked __noinline __used static int ptr_stack_callee(void)
+{
+	asm volatile ("					\
+	r6 = r10;					\
+	r6 += -16;					\
+	r7 = r1;					\
+	r0 = 0;						\
+1:	r6 += 1;					\
+	r0 += 1;					\
+	if r0 & 8 goto 4f;	/* bound main pass iteration, but hide it from SCEV */	\
+	if r6 < r7 goto 1b;	/* r6 and r7 are from different frames, can't be widened */	\
+4:	r0 = 0;						\
+	exit;						\
+"	::: __clobber_all);
+}
+
+/* The end pointer belongs to the caller's frame. */
+SEC("xdp")
+__success
+__log_level(2)
+__flag(BPF_F_TEST_STATE_FREQ)
+__msg("scev: incompatible latch operands r6 (fp), r7 (fp)")
+__not_msg("widening r6")
+__naked void ptr_stack_other_frame(void)
+{
+	asm volatile ("					\
+	r1 = r10;					\
+	r1 += -12;					\
+	call ptr_stack_callee;				\
+	exit;						\
+"	::: __clobber_all);
+}
+
+/* Metadata and packet data have different origins. */
+SEC("xdp")
+__success
+__log_level(2)
+__flag(BPF_F_TEST_STATE_FREQ)
+__msg("scev: incompatible latch operands r6 (pkt_meta), r7 (pkt)")
+__not_msg("widening r6")
+__naked void ptr_packet_meta_other_origin(void)
+{
+	asm volatile ("					\
+	r6 = *(u32 *)(r1 + %[data_meta]);		\
+	r7 = *(u32 *)(r1 + %[data]);			\
+	r7 += 4;					\
+	r0 = 0;						\
+1:	r6 += 1;					\
+	r0 += 1;					\
+	if r0 & 8 goto 4f;				\
+	if r6 < r7 goto 1b;				\
+4:	r0 = 0;						\
+	exit;						\
+"	:
+	: __imm_const(data, offsetof(struct xdp_md, data)),
+	  __imm_const(data_meta, offsetof(struct xdp_md, data_meta))
+	: __clobber_all);
+}
+
+/* Two lookups in a 1-element array count as the same memory origin. */
+SEC("xdp")
+__success
+__log_level(2)
+__flag(BPF_F_TEST_STATE_FREQ)
+__msg("header_count is 4 ")
+__msg("widening r6 to 0..3 step 1")
+__naked void ptr_map_value_two_lookups(void)
+{
+	asm volatile ("					\
+	*(u64 *)(r10 - 8) = 0;				\
+	r2 = r10;					\
+	r2 += -8;					\
+	r1 = %[small_map] ll;				\
+	call %[bpf_map_lookup_elem];			\
+	if r0 == 0 goto 2f;				\
+	r6 = r0;					\
+	r2 = r10;					\
+	r2 += -8;					\
+	r1 = %[small_map] ll;				\
+	call %[bpf_map_lookup_elem];			\
+	if r0 == 0 goto 2f;				\
+	r7 = r0;					\
+	r7 += 4;					\
+1:	r6 += 1;					\
+	if r6 < r7 goto 1b;				\
+2:	r0 = 0;						\
+	exit;						\
+"	:
+	: __imm(bpf_map_lookup_elem),
+	  __imm_addr(small_map)
+	: __clobber_all);
+}
+
+/*
+ * For inner array maps with BPF_F_INNER_MAP flag the actual array size
+ * might differ from metadata in map declaration. Hence, r6 and r7 below
+ * cannot be directly compared / iterations count can't be inferred.
+ *
+ *   r8 = bpf_map_lookup_elem(outer_map, ...); // inner map
+ *   r6 = bpf_map_lookup_elem(r8, ...);
+ *   r7 = bpf_map_lookup_elem(r8, ...);
+ *   r7 += 4;
+ *   do { r6++; ... } while (r6 < r7);
+ */
+SEC("xdp")
+__success
+__log_level(2)
+__flag(BPF_F_TEST_STATE_FREQ)
+__msg("scev: incompatible latch operands r6 (map_value), r7 (map_value)")
+__not_msg("widening r6")
+__naked void ptr_map_value_two_lookups_inner_map(void)
+{
+	asm volatile ("					\
+	*(u64 *)(r10 - 8) = 0;				\
+	r2 = r10;					\
+	r2 += -8;					\
+	r1 = %[outer_map] ll;				\
+	call %[bpf_map_lookup_elem];			\
+	if r0 == 0 goto 2f;				\
+	r8 = r0;					\
+	r2 = r10;					\
+	r2 += -8;					\
+	r1 = r8;					\
+	call %[bpf_map_lookup_elem];			\
+	if r0 == 0 goto 2f;				\
+	r6 = r0;					\
+	r1 = r8;					\
+	r2 = r10;					\
+	r2 += -8;					\
+	call %[bpf_map_lookup_elem];			\
+	if r0 == 0 goto 2f;				\
+	r7 = r0;					\
+	r7 += 4;					\
+	r0 = 0;						\
+1:	r6 += 1;					\
+	r0 += 1;					\
+	if r0 & 8 goto 2f;	/* main pass iteration guard */ \
+	if r6 < r7 goto 1b;				\
+2:	r0 = 0;						\
+	exit;						\
+"	:
+	: __imm(bpf_map_lookup_elem),
+	  __imm_addr(outer_map)
+	: __clobber_all);
+}
+
+SEC("xdp")
+__success
+__log_level(2)
+__flag(BPF_F_TEST_STATE_FREQ)
+__msg("header_count is 4 ")
+__msg("widening r6 to 0..3 step 1")
+__naked void ptr_map_value(void)
+{
+	asm volatile ("					\
+	*(u64 *)(r10 - 8) = 0;				\
+	r2 = r10;					\
+	r2 += -8;					\
+	r1 = %[map] ll;					\
+	call %[bpf_map_lookup_elem];			\
+	if r0 == 0 goto 2f;				\
+	r6 = r0;					\
+	r7 = r6;					\
+	r7 += 4;					\
+1:	r6 += 1;					\
+	if r6 < r7 goto 1b;				\
+2:	r0 = 0;						\
+	exit;						\
+"	:
+	: __imm(bpf_map_lookup_elem), __imm_addr(map)
+	: __clobber_all);
+}
+
+/* Equal offsets into different maps do not establish a common origin. */
+SEC("xdp")
+__success
+__log_level(2)
+__flag(BPF_F_TEST_STATE_FREQ)
+__msg("scev: incompatible latch operands r6 (map_value), r7 (map_value)")
+__not_msg("widening r6")
+__naked void ptr_map_value_other_map(void)
+{
+	asm volatile ("					\
+	*(u64 *)(r10 - 8) = 0;				\
+	r2 = r10;					\
+	r2 += -8;					\
+	r1 = %[map] ll;					\
+	call %[bpf_map_lookup_elem];			\
+	if r0 == 0 goto 2f;				\
+	r6 = r0;					\
+	r2 = r10;					\
+	r2 += -8;					\
+	r1 = %[other_map] ll;				\
+	call %[bpf_map_lookup_elem];			\
+	if r0 == 0 goto 2f;				\
+	r7 = r0;					\
+	r7 += 4;					\
+	r0 = 0;						\
+1:	r6 += 1;					\
+	r0 += 1;					\
+	if r0 & 8 goto 4f;				\
+	if r6 < r7 goto 1b;				\
+4:							\
+2:	r0 = 0;						\
+	exit;						\
+"	:
+	: __imm(bpf_map_lookup_elem), __imm_addr(map), __imm_addr(other_map)
+	: __clobber_all);
+}
+
+SEC("xdp")
+__success
+__log_level(2)
+__flag(BPF_F_TEST_STATE_FREQ)
+__msg("header_count is 4 ")
+__msg("widening r6 to 0..3 step 1")
+__naked void ptr_mem(void)
+{
+	asm volatile ("					\
+	r1 = %[ringbuf] ll;				\
+	r2 = 8;						\
+	r3 = 0;						\
+	call %[bpf_ringbuf_reserve];			\
+	if r0 == 0 goto 2f;				\
+	r8 = r0;					\
+	r6 = r0;					\
+	r7 = r6;					\
+	r7 += 4;					\
+1:	r6 += 1;					\
+	if r6 < r7 goto 1b;				\
+	r1 = r8;					\
+	r2 = 0;						\
+	call %[bpf_ringbuf_discard];			\
+2:	r0 = 0;						\
+	exit;						\
+"	:
+	: __imm(bpf_ringbuf_reserve), __imm(bpf_ringbuf_discard), __imm_addr(ringbuf)
+	: __clobber_all);
+}
+
+/* Separate reservations have different pointer IDs. */
+SEC("xdp")
+__success
+__log_level(2)
+__flag(BPF_F_TEST_STATE_FREQ)
+__msg("scev: incompatible latch operands r6 (ringbuf_mem), r7 (ringbuf_mem)")
+__not_msg("widening r6")
+__naked void ptr_mem_other_reservation(void)
+{
+	asm volatile ("					\
+	r1 = %[ringbuf] ll;				\
+	r2 = 8;						\
+	r3 = 0;						\
+	call %[bpf_ringbuf_reserve];			\
+	if r0 == 0 goto 3f;				\
+	r8 = r0;					\
+	r6 = r0;					\
+	r1 = %[ringbuf] ll;				\
+	r2 = 8;						\
+	r3 = 0;						\
+	call %[bpf_ringbuf_reserve];			\
+	if r0 == 0 goto 2f;				\
+	r9 = r0;					\
+	r7 = r0;					\
+	r7 += 4;					\
+	r0 = 0;						\
+1:	r6 += 1;					\
+	r0 += 1;					\
+	if r0 & 8 goto 4f;				\
+	if r6 < r7 goto 1b;				\
+4:	r1 = r9;					\
+	r2 = 0;						\
+	call %[bpf_ringbuf_discard];			\
+2:	r1 = r8;					\
+	r2 = 0;						\
+	call %[bpf_ringbuf_discard];			\
+3:	r0 = 0;						\
+	exit;						\
+"	:
+	: __imm(bpf_ringbuf_reserve), __imm(bpf_ringbuf_discard), __imm_addr(ringbuf)
+	: __clobber_all);
+}
+
+SEC("iter/bpf_map_elem")
+__success
+__log_level(2)
+__flag(BPF_F_TEST_STATE_FREQ)
+__msg("header_count is 4 ")
+__msg("widening r6 to 0..3 step 1")
+__naked void ptr_buf(void)
+{
+	asm volatile ("					\
+	r6 = *(u64 *)(r1 + %[value]);			\
+	if r6 == 0 goto 2f;				\
+	r7 = r6;					\
+	r7 += 4;					\
+1:	r6 += 1;					\
+	if r6 < r7 goto 1b;				\
+2:	r0 = 0;						\
+	exit;						\
+"	:
+	: __imm_const(value, offsetof(struct bpf_iter__bpf_map_elem, value))
+	: __clobber_all);
+}
+
+/* Separate context loads receive different pointer IDs. */
+SEC("iter/bpf_map_elem")
+__success
+__log_level(2)
+__flag(BPF_F_TEST_STATE_FREQ)
+__msg("scev: incompatible latch operands r6 (buf), r7 (buf)")
+__not_msg("widening r6")
+__naked void ptr_buf_other_load(void)
+{
+	asm volatile ("					\
+	r6 = *(u64 *)(r1 + %[value]);			\
+	if r6 == 0 goto 2f;				\
+	r7 = *(u64 *)(r1 + %[value]);			\
+	if r7 == 0 goto 2f;				\
+	r7 += 4;					\
+	r0 = 0;						\
+1:	r6 += 1;					\
+	r0 += 1;					\
+	if r0 & 8 goto 4f;				\
+	if r6 < r7 goto 1b;				\
+4:							\
+2:	r0 = 0;						\
+	exit;						\
+"	:
+	: __imm_const(value, offsetof(struct bpf_iter__bpf_map_elem, value))
+	: __clobber_all);
+}
+
+/* A pointer-versus-scalar latch must not establish an iteration count. */
+SEC("xdp")
+__success
+__log_level(2)
+__flag(BPF_F_TEST_STATE_FREQ)
+__msg("scev: incompatible latch operands r1 (map_value), scalar 4")
+__not_msg("widening r1")
+__naked void ptr_vs_scalar_no_widen(void)
+{
+	asm volatile ("					\
+	*(u64 *)(r10 - 8) = 0;				\
+	r2 = r10;					\
+	r2 += -8;					\
+	r1 = %[map] ll;					\
+	call %[bpf_map_lookup_elem];			\
+	if r0 == 0 goto out_%=;				\
+	r1 = r0;					\
+	r0 = 0;						\
+loop_%=:						\
+	r1 += 1;					\
+	r0 += 1;					\
+	if r0 & 8 goto out_%=;				\
+	if r1 < 4 goto loop_%=;				\
+out_%=:							\
+	r0 = 0;						\
+	exit;						\
+"	:
+	: __imm(bpf_map_lookup_elem),
+	  __imm_addr(map)
+	: __clobber_all);
+}
+
+/* A nullable pointer must not be widened before its null check. */
+SEC("xdp")
+__success
+__log_level(2)
+__flag(BPF_F_TEST_STATE_FREQ)
+__msg("loop header at 9, can't widen r1, expr is (+ r1 8)")
+__naked void no_widen_maybe_null(void)
+{
+	asm volatile ("					\
+	r1 = 0;						\
+	*(u64 *)(r10 - 8) = r1;				\
+	r2 = r10;					\
+	r2 += -8;					\
+	r1 = %[map] ll;					\
+	call %[bpf_map_lookup_elem];			\
+	r1 = r0;					\
+	r6 = 0;						\
+loop_%=:						\
+	if r1 == 0 goto exit_%=;			\
+	r1 += 8;					\
+	r6 += 1;					\
+	if r6 < 3 goto loop_%=;				\
+exit_%=:						\
+	r0 = 0;						\
+	exit;						\
+"	:
+	: __imm(bpf_map_lookup_elem),
+	  __imm_addr(map)
+	: __clobber_all);
+}
+
+/*
+ * A terminating inner loop does not prove termination of the outer loop.
+ *
+ *    u64 r7 = 0;
+ *    while (bpf_get_prandom_u32() != 42) {
+ *        r7 = (r7 + 1) & 0xf;
+ *        u64 r8 = 0;
+ *        do {
+ *            r8++;
+ *        } while (r8 < 3);
+ *    }
+ *    return 0;
+ */
+SEC("xdp")
+__failure
+__log_level(2)
+__flag(BPF_F_TEST_STATE_FREQ)
+__msg("infinite loop detected at insn 6")
+__naked void nested_terminating_prune(void)
+{
+	asm volatile ("					\
+	r7 = 0;						\
+outer_%=:						\
+	r7 += 1;					\
+	r7 &= 0xf;					\
+	call %[bpf_get_prandom_u32];			\
+	if r0 == 42 goto out_%=;			\
+	r8 = 0;						\
+inner_%=:						\
+	r8 += 1;					\
+	if r8 < 3 goto inner_%=;			\
+	goto outer_%=;					\
+out_%=:							\
+	r0 = 0;						\
+	exit;						\
+"	:
+	: __imm(bpf_get_prandom_u32)
+	: __clobber_all);
+}
+
+/*
+ * Returning directly from the inner loop to the outer header changes the
+ * active stack from [1, 4] to [1].
+ */
+SEC("xdp")
+__success
+__log_level(2)
+__flag(BPF_F_TEST_STATE_FREQ)
+__msg("loop at 1{{$}}")
+__msg("loop at 4, nested in 1")
+__msg("entering loop 1")
+__msg("entering loop 4")
+__msg("exiting loop 4")
+__not_msg("entering loop 1")
+__msg("from 4 to 1:")
+__msg("loop_stack=1{{$}}")
+__naked void nested_loop_backedge_to_outer_header(void)
+{
+	asm volatile ("					\
+	r6 = 0;						\
+outer_%=:						\
+	if r6 == 3 goto exit_%=;			\
+	r6 += 1;					\
+	r7 = 0;						\
+inner_%=:						\
+	if r7 == 2 goto outer_%=;			\
+	r7 += 1;					\
+	goto inner_%=;					\
+exit_%=:						\
+	r0 = 0;						\
+	exit;						\
+"	::: __clobber_all);
+}
+
+/* Induction variable as the src operand of the latch compare. */
+SEC("socket")
+__success
+__log_level(2)
+__msg("loop header at 2, header_count is 10")
+__msg("loop header at 2, widening r1 to 0..9 step 1")
+__flag(BPF_F_TEST_STATE_FREQ)
+__naked void post_cond_ivar_src(void)
+{
+	asm volatile ("					\
+	r1 = 0;						\
+	r2 = 10;					\
+1:	r1 += 1;					\
+	if r2 != r1 goto 1b;				\
+	r0 = 0;						\
+	exit;						\
+"	::: __clobber_all);
+}
+
+/* Early exit dominates the latch and gives a tighter bound. */
+SEC("socket")
+__success
+__log_level(2)
+__msg("  backedge from 4, latch at 4, dominating exits at 2")
+__msg("loop header at 2, header_count is [0..15]")
+__msg("loop header at 2, widening r1 to 0..14 step 1")
+__flag(BPF_F_TEST_STATE_FREQ)
+__naked void dominating_exit_bounds_count(void)
+{
+	asm volatile ("					\
+	r1 = 0;						\
+	r2 = 14;					\
+1:	if r2 == r1 goto 2f;				\
+	r1 += 1;					\
+	if r1 != 63 goto 1b;				\
+2:	r0 = 0;						\
+	exit;						\
+"	::: __clobber_all);
+}
+
+char _license[] SEC("license") = "GPL";

-- 
2.53.0

Re: [PATCH bpf-next v3 51/51] selftests/bpf: tests for SCEV analysis and loop widening

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 1433 bytes --]

> selftests/bpf: tests for SCEV analysis and loop widening

TL;DR: Nit only, in the selftest: nested_loop1() and expr_chain() pass an
unused __imm(bpf_get_prandom_u32) operand that could be dropped.

> diff --git a/tools/testing/selftests/bpf/progs/verifier_scev.c b/tools/testing/selftests/bpf/progs/verifier_scev.c
> new file mode 100644
> index 000000000000..59c4b2864db0
> --- /dev/null
> +++ b/tools/testing/selftests/bpf/progs/verifier_scev.c

[ ... ]

> +__naked void nested_loop1(void)
> +{
> +	asm volatile ("					\
> +	r6 = 0;						\
> +1:							\
> +	if r6 == 2 goto 2f;				\
> +	r6 += 1;					\
> +	r7 = 0;						\
> +3:							\
> +	if r7 == 2 goto 4f;				\
> +	r7 += 1;					\
> +	goto 3b;					\
> +4:							\
> +	goto 1b;					\
> +2:							\
> +	r0 = r7;					\
> +	exit;						\
> +"	:
> +	: __imm(bpf_get_prandom_u32)
> +	: __clobber_all);
> +}

This isn't a bug, but nested_loop1() never calls bpf_get_prandom_u32.  Could
the __imm(bpf_get_prandom_u32) operand be dropped here, and in expr_chain()
too, so the asm uses ": : : __clobber_all" like the neighbouring loop tests?

The tests that do call the helper (meet_agrees, meet_disagrees) need the
operand, but these two do not.


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/37923187511

Re: [PATCH bpf-next v3 00/51] bpf: use scalar evolution to widen bounded loops

[patchwork-bot+netdevbpf]

Hello:

This series was applied to bpf/bpf-next.git (master)
by Alexei Starovoitov <ast@kernel.org>:

On Fri,  9 Oct 2026 03:47:18 -0700 you wrote:
> This series implements the scalar evolution (SCEV) technique for
> verification of loops.
> 
> Scalar evolution is a static analysis technique that infers algebraic
> expressions describing how variables change within a loop body.
> These expressions can then be used to estimate the number of loop body
> executions. This estimate can be used to represent induction variables
> as ranges instead of enumerating each possible value.
> 
> [...]

Here is the summary with links:
  - [bpf-next,v3,01/51] bpf: represent stack access effects with arg_access_info
    https://git.kernel.org/bpf/bpf-next/c/5b6beb95f7b4
  - [bpf-next,v3,02/51] bpf: describe helper stack accesses with arg_access_info
    https://git.kernel.org/bpf/bpf-next/c/830e4372de5c
  - [bpf-next,v3,03/51] bpf: describe kfunc stack accesses with arg_access_info
    https://git.kernel.org/bpf/bpf-next/c/8f3bd4537faa
  - [bpf-next,v3,04/51] bpf: track may_write flags in liveness
    https://git.kernel.org/bpf/bpf-next/c/ac43e0393f44
  - [bpf-next,v3,05/51] bpf: summarize may write stack slots in insn_aux_data
    https://git.kernel.org/bpf/bpf-next/c/0360ed8655cf
  - [bpf-next,v3,06/51] bpf: summarize live stack slots in insn_aux_data
    https://git.kernel.org/bpf/bpf-next/c/30175f5d9b1a
  - [bpf-next,v3,07/51] bpf: summarize regs that may hold a frame pointer in insn_aux_data
    https://git.kernel.org/bpf/bpf-next/c/c6477be15804
  - [bpf-next,v3,08/51] bpf: record write effects for atomic operations in liveness.c
    https://git.kernel.org/bpf/bpf-next/c/2d903f95721a
  - [bpf-next,v3,09/51] bpf: fix arg_track_xfer() for atomic stack spills and fills
    https://git.kernel.org/bpf/bpf-next/c/13cffb883180
  - [bpf-next,v3,10/51] bpf: derive global subprog stack liveness from BTF signatures
    https://git.kernel.org/bpf/bpf-next/c/8212c1c2986d
  - [bpf-next,v3,11/51] bpf: compute parameters liveness in bpf_compute_live_registers()
    https://git.kernel.org/bpf/bpf-next/c/147400e9c27c
  - [bpf-next,v3,12/51] bpf: add tnum_alignment()
    https://git.kernel.org/bpf/bpf-next/c/4072d7f56b44
  - [bpf-next,v3,13/51] bpf: add cnum{32,64}_union()
    https://git.kernel.org/bpf/bpf-next/c/a8e9a53f6df6
  - [bpf-next,v3,14/51] bpf: add cnum64_intersect_linear()
    https://git.kernel.org/bpf/bpf-next/c/78e61c01a760
  - [bpf-next,v3,15/51] bpf: expose bpf_rev_opcode() and bpf_flip_opcode()
    https://git.kernel.org/bpf/bpf-next/c/685ac494443e
  - [bpf-next,v3,16/51] bpf: expose bpf_grow_stack_state()
    https://git.kernel.org/bpf/bpf-next/c/041d7c85c45d
  - [bpf-next,v3,17/51] bpf: expose bpf_verbose_snum()
    https://git.kernel.org/bpf/bpf-next/c/f21dcf26986c
  - [bpf-next,v3,18/51] bpf: allow subrange relations for PTR_TO_STACK in regsafe()
    https://git.kernel.org/bpf/bpf-next/c/366b4c68d830
  - [bpf-next,v3,19/51] bpf: representation for intervals with steps
    https://git.kernel.org/bpf/bpf-next/c/0892ef2a7ef3
  - [bpf-next,v3,20/51] bpf: varying offset access support for PTR_TO_BTF_ID pointers
    https://git.kernel.org/bpf/bpf-next/c/89e72f81a455
  - [bpf-next,v3,21/51] bpf: save DFS postorder numbers for program instructions
    https://git.kernel.org/bpf/bpf-next/c/f414d20879ae
  - [bpf-next,v3,22/51] bpf: move the live-register and SCC printout to a standalone function
    https://git.kernel.org/bpf/bpf-next/c/ceea9085f2d1
  - [bpf-next,v3,23/51] bpf: compute immediate dominators
    https://git.kernel.org/bpf/bpf-next/c/6254fb693289
  - [bpf-next,v3,24/51] bpf: compute loop hierarchy
    https://git.kernel.org/bpf/bpf-next/c/9ae7f5e8bf1d
  - [bpf-next,v3,25/51] bpf: add bpf_set_reg_range()
    https://git.kernel.org/bpf/bpf-next/c/f849c3a145eb
  - [bpf-next,v3,26/51] bpf: add bpf_mark_reg_known_scalar()
    https://git.kernel.org/bpf/bpf-next/c/bc2afa09c07e
  - [bpf-next,v3,27/51] bpf: add bpf_reg_union() and bpf_reg_union_compatible()
    https://git.kernel.org/bpf/bpf-next/c/6360b15f86ed
  - [bpf-next,v3,28/51] bpf: add bpf_split_cur_state()
    https://git.kernel.org/bpf/bpf-next/c/97d592e69d09
  - [bpf-next,v3,29/51] bpf: add bpf_same_memory_origin()
    https://git.kernel.org/bpf/bpf-next/c/4c3a56544b65
  - [bpf-next,v3,30/51] bpf: add bpf_spill_size()
    https://git.kernel.org/bpf/bpf-next/c/1291ad590693
  - [bpf-next,v3,31/51] bpf: record basic-block ends in insn_aux_data
    https://git.kernel.org/bpf/bpf-next/c/2fa579694ddf
  - [bpf-next,v3,32/51] bpf: allow precision backtracking between overlapping checkpoints
    https://git.kernel.org/bpf/bpf-next/c/60b0109ad195
  - [bpf-next,v3,33/51] bpf: compute scalar evolution expressions for loops
    https://git.kernel.org/bpf/bpf-next/c/1501b536818b
  - [bpf-next,v3,34/51] bpf: use SCEV to widen bounded loops
    https://git.kernel.org/bpf/bpf-next/c/6ccf49f79327
  - [bpf-next,v3,35/51] bpf: avoid widening registers that hinder exact stack-slot tracking
    https://git.kernel.org/bpf/bpf-next/c/088a5477a9bd
  - [bpf-next,v3,36/51] bpf: bpf_func_state size optimization
    https://git.kernel.org/bpf/bpf-next/c/a2052333e6f9
  - [bpf-next,v3,37/51] selftests/bpf: __msg_next tag for matching messages on consecutive lines
    https://git.kernel.org/bpf/bpf-next/c/9fb8927b956b
  - [bpf-next,v3,38/51] selftests/bpf: bound UNIX socket path loops by sun_path size
    https://git.kernel.org/bpf/bpf-next/c/0c7530110d27
  - [bpf-next,v3,39/51] selftests/bpf: test for stack-pointer subrange pruning
    https://git.kernel.org/bpf/bpf-next/c/73e7927bdafa
  - [bpf-next,v3,40/51] selftests/bpf: tests for may_write stack-liveness tracking
    https://git.kernel.org/bpf/bpf-next/c/c4db12b80c09
  - [bpf-next,v3,41/51] selftests/bpf: tests for may_def marks of atomic RMW operations
    https://git.kernel.org/bpf/bpf-next/c/2c719872958c
  - [bpf-next,v3,42/51] selftests/bpf: tests for stack pointer spills/fills through atomics
    https://git.kernel.org/bpf/bpf-next/c/75a46f59e098
  - [bpf-next,v3,43/51] selftests/bpf: tests for map special cases in bpf_helper_stack_access_bytes()
    https://git.kernel.org/bpf/bpf-next/c/e60fb9b6b5be
  - [bpf-next,v3,44/51] selftests/bpf: tests for BTF-based global subprog stack liveness
    https://git.kernel.org/bpf/bpf-next/c/063c85f5d8c7
  - [bpf-next,v3,45/51] selftests/bpf: test for register liveness of subprogram arguments
    https://git.kernel.org/bpf/bpf-next/c/a88078acc2e4
  - [bpf-next,v3,46/51] selftests/bpf: tests for register base/step arithmetic
    https://git.kernel.org/bpf/bpf-next/c/2db5837f9ada
  - [bpf-next,v3,47/51] selftests/bpf: tests for register base/step state pruning
    https://git.kernel.org/bpf/bpf-next/c/8c5f6dfefdf1
  - [bpf-next,v3,48/51] selftests/bpf: tests for varying offset access to PTR_TO_BTF_ID
    https://git.kernel.org/bpf/bpf-next/c/e187fac8ac6b
  - [bpf-next,v3,49/51] selftests/bpf: tests for loop hierarchy computation
    https://git.kernel.org/bpf/bpf-next/c/951538da0fa9
  - [bpf-next,v3,50/51] selftests/bpf: tests for immediate dominator computation
    https://git.kernel.org/bpf/bpf-next/c/aa8964a8e5f1
  - [bpf-next,v3,51/51] selftests/bpf: tests for SCEV analysis and loop widening
    https://git.kernel.org/bpf/bpf-next/c/8b93e9f92c16

You are awesome, thank you!
-- 
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html