[PATCH bpf-next v3 0/6] bpf: Inline the numeric open-coded iterator kfuncs

[Puranjay Mohan <puranjay@kernel.org>]

The bpf_for(i, start, end) macro is BPF's open-coded numeric iterator. It
expands into calls to three kfuncs: bpf_iter_num_new() to set the iterator
up, bpf_iter_num_next() once per iteration, and bpf_iter_num_destroy() to
tear it down. The verifier emits these as ordinary kfunc calls, so a
bpf_for() loop pays function-call overhead on setup, teardown, and — most
importantly — on every single iteration via bpf_iter_num_next().

All three kfuncs are tiny and only touch the 8-byte on-stack iterator state
(struct bpf_iter_num_kern { int cur; int end; }). That makes them good
candidates for inlining, the same way several other special kfuncs are
already open-coded in bpf_fixup_kfunc_call(). This series replaces each of
the three calls with an equivalent inline BPF instruction sequence:

  - bpf_iter_num_new(): the (s64)end - (s64)start overflow check is done
    with 32-bit arithmetic (start <= end is checked first, so the range
    fits in a u32), avoiding cpuv4 sign-extension insns that some JITs do
    not implement. Returns the same -EINVAL / -E2BIG / 0 as the kfunc.

  - bpf_iter_num_next(): the hot path. Since cur and end are int, the
    kfunc's (s64)(s->cur + 1) >= s->end test reduces to a signed 32-bit
    comparison of (s->cur + 1) against s->end, so the inlined code uses a
    32-bit compare with no sign extension.

  - bpf_iter_num_destroy(): a single 8-byte store zeroing the state.

bpf_for() is frequently used with constant bounds; in that case the range
checks in bpf_iter_num_new() are decidable at verification time, so a
follow-up patch elides them and emits only the iterator init (or the error
result). The inlined shapes are pinned by new __xlated selftests, and a
bench_bpf_for benchmark (modeled on the existing bpf_loop benchmark) runs a
bpf_for() loop with an empty body to measure the per-iteration cost.

The emitted instructions are plain BPF and remain valid for the
interpreter, so interpreter fallback stays correct and no jit_required
marking is needed.

Benchmark (./bench -p 1 --nr_loops 1000000 {bpf-loop,bpf-for}):

    +--------+---------------------+---------------------+---------------------+
    |  arch  |       bpf_loop      | bpf_for non-inlined |   bpf_for inlined   |
    +--------+---------------------+---------------------+---------------------+
    | x86-64 | 2946 M/s (0.339 ns) | 4281 M/s (0.234 ns) | 8981 M/s (0.111 ns) |
    +--------+---------------------+---------------------+---------------------+
    | arm64  |  618 M/s (1.619 ns) |  543 M/s (1.843 ns) |  538 M/s (1.858 ns) |
    +--------+---------------------+---------------------+---------------------+

On x86-64, removing the per-iteration call to bpf_iter_num_next() roughly
doubles bpf_for() throughput. On arm64 it is neutral: the loop is bound by
the load/store dependency chain on the on-stack iterator state rather than
by call overhead, so inlining neither helps nor hurts there. It still
removes the calls.

bpf_loop() is shown for reference only; it is a different construct (a
callback invoked per iteration), so the comparison is structural rather
than a measure of the inlining: on x86-64 bpf_for() is faster than
bpf_loop() even before inlining, while on arm64 bpf_loop() is faster
because bpf_for()'s per-iteration cost is dominated by the stack round-trip
through the iterator state.

Changelog:
v2: https://lore.kernel.org/bpf/20260717120215.2171057-1-puranjay@kernel.org/
Changes in v3:
- Elide the range checks in bpf_iter_num_new() when start and end are
  constant, marking the registers precise so paths reaching the call with
  different constants are not pruned (Eduard Zingerman)
- Add __xlated selftests pinning the inlined new()/next()/destroy() shapes
  (Eduard Zingerman)
- Use the insn_buf[i++] idiom in the inline helpers (Eduard Zingerman)
- Pick up Acked-by on patch 3

v1: https://lore.kernel.org/all/20260715130430.318421-1-puranjay@kernel.org/
Changes in v2:
- Don't emit sign-extending (movsx) moves; some JITs (e.g. x86-32, mips32,
  sparc64) decode them as a plain move and would miscompile the range check

Puranjay Mohan (6):
  bpf: Inline bpf_iter_num_new() kfunc
  bpf: Inline bpf_iter_num_next() kfunc
  bpf: Inline bpf_iter_num_destroy() kfunc
  bpf: Elide range checks when inlining bpf_iter_num_new() for constant
    bounds
  selftests/bpf: Verify inlined numeric iterator shape with __xlated
  selftests/bpf: Add bpf_for() benchmark

 include/linux/bpf_verifier.h                  |  13 ++
 kernel/bpf/verifier.c                         | 157 ++++++++++++++++++
 tools/testing/selftests/bpf/Makefile          |   2 +
 tools/testing/selftests/bpf/bench.c           |   4 +
 .../selftests/bpf/benchs/bench_bpf_for.c      | 104 ++++++++++++
 .../selftests/bpf/benchs/run_bench_bpf_for.sh |  15 ++
 .../selftests/bpf/progs/bpf_for_bench.c       |  32 ++++
 tools/testing/selftests/bpf/progs/iters.c     | 145 ++++++++++++++++
 8 files changed, 472 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/benchs/bench_bpf_for.c
 create mode 100755 tools/testing/selftests/bpf/benchs/run_bench_bpf_for.sh
 create mode 100644 tools/testing/selftests/bpf/progs/bpf_for_bench.c


base-commit: a23a71823352e2d792dcaae25f1ebb744acbfc0b
-- 
2.53.0-Meta