Re: [PATCH bpf-next v3 0/6] bpf: Inline the numeric open-coded iterator kfuncs

["Kumar Kartikeya Dwivedi" <memxor@gmail.com>]

On Wed Jul 22, 2026 at 3:24 PM CEST, Puranjay Mohan wrote:
> The bpf_for(i, start, end) macro is BPF's open-coded numeric iterator. It
> expands into calls to three kfuncs: bpf_iter_num_new() to set the iterator
> up, bpf_iter_num_next() once per iteration, and bpf_iter_num_destroy() to
> tear it down. The verifier emits these as ordinary kfunc calls, so a
> bpf_for() loop pays function-call overhead on setup, teardown, and — most
> importantly — on every single iteration via bpf_iter_num_next().
>
> All three kfuncs are tiny and only touch the 8-byte on-stack iterator state
> (struct bpf_iter_num_kern { int cur; int end; }). That makes them good
> candidates for inlining, the same way several other special kfuncs are
> already open-coded in bpf_fixup_kfunc_call(). This series replaces each of
> the three calls with an equivalent inline BPF instruction sequence:
>
>   - bpf_iter_num_new(): the (s64)end - (s64)start overflow check is done
>     with 32-bit arithmetic (start <= end is checked first, so the range
>     fits in a u32), avoiding cpuv4 sign-extension insns that some JITs do
>     not implement. Returns the same -EINVAL / -E2BIG / 0 as the kfunc.
>
>   - bpf_iter_num_next(): the hot path. Since cur and end are int, the
>     kfunc's (s64)(s->cur + 1) >= s->end test reduces to a signed 32-bit
>     comparison of (s->cur + 1) against s->end, so the inlined code uses a
>     32-bit compare with no sign extension.
>
>   - bpf_iter_num_destroy(): a single 8-byte store zeroing the state.
>
> bpf_for() is frequently used with constant bounds; in that case the range
> checks in bpf_iter_num_new() are decidable at verification time, so a
> follow-up patch elides them and emits only the iterator init (or the error
> result). The inlined shapes are pinned by new __xlated selftests, and a
> bench_bpf_for benchmark (modeled on the existing bpf_loop benchmark) runs a
> bpf_for() loop with an empty body to measure the per-iteration cost.
>
> The emitted instructions are plain BPF and remain valid for the
> interpreter, so interpreter fallback stays correct and no jit_required
> marking is needed.
>
> Benchmark (./bench -p 1 --nr_loops 1000000 {bpf-loop,bpf-for}):
>
>     +--------+---------------------+---------------------+---------------------+
>     |  arch  |       bpf_loop      | bpf_for non-inlined |   bpf_for inlined   |
>     +--------+---------------------+---------------------+---------------------+
>     | x86-64 | 2946 M/s (0.339 ns) | 4281 M/s (0.234 ns) | 8981 M/s (0.111 ns) |
>     +--------+---------------------+---------------------+---------------------+
>     | arm64  |  618 M/s (1.619 ns) |  543 M/s (1.843 ns) |  538 M/s (1.858 ns) |
>     +--------+---------------------+---------------------+---------------------+
>
> On x86-64, removing the per-iteration call to bpf_iter_num_next() roughly
> doubles bpf_for() throughput. On arm64 it is neutral: the loop is bound by
> the load/store dependency chain on the on-stack iterator state rather than
> by call overhead, so inlining neither helps nor hurts there. It still
> removes the calls.

I find some of this explanation unsatisfying. How was this determined? Is this
just your hypothesis, or did you actually observe different counters in perf
state to corroborate this on arm64? I am no arm64 expert, so perhaps it does not
do a bunch of things (store-load forwarding and memory disambiguation) that
would help here, but it would still be good to understand why.

It would also be interesting to compare against can_loop. Adding it as a
comparison and finding the result should be easy. Can you check what the
overhead is in that case?

All that said, I am still fine with all this, bpf_for() being costly has been a
known fact forever.

>
> bpf_loop() is shown for reference only; it is a different construct (a
> callback invoked per iteration), so the comparison is structural rather
> than a measure of the inlining: on x86-64 bpf_for() is faster than
> bpf_loop() even before inlining, while on arm64 bpf_loop() is faster
> because bpf_for()'s per-iteration cost is dominated by the stack round-trip
> through the iterator state.
>
> Changelog:
> v2: https://lore.kernel.org/bpf/20260717120215.2171057-1-puranjay@kernel.org/
> Changes in v3:
> - Elide the range checks in bpf_iter_num_new() when start and end are
>   constant, marking the registers precise so paths reaching the call with
>   different constants are not pruned (Eduard Zingerman)
> - Add __xlated selftests pinning the inlined new()/next()/destroy() shapes
>   (Eduard Zingerman)
> - Use the insn_buf[i++] idiom in the inline helpers (Eduard Zingerman)
> - Pick up Acked-by on patch 3
>
> v1: https://lore.kernel.org/all/20260715130430.318421-1-puranjay@kernel.org/
> Changes in v2:
> - Don't emit sign-extending (movsx) moves; some JITs (e.g. x86-32, mips32,
>   sparc64) decode them as a plain move and would miscompile the range check
>
> Puranjay Mohan (6):
>   bpf: Inline bpf_iter_num_new() kfunc
>   bpf: Inline bpf_iter_num_next() kfunc
>   bpf: Inline bpf_iter_num_destroy() kfunc
>   bpf: Elide range checks when inlining bpf_iter_num_new() for constant
>     bounds
>   selftests/bpf: Verify inlined numeric iterator shape with __xlated
>   selftests/bpf: Add bpf_for() benchmark
>
>  include/linux/bpf_verifier.h                  |  13 ++
>  kernel/bpf/verifier.c                         | 157 ++++++++++++++++++
>  tools/testing/selftests/bpf/Makefile          |   2 +
>  tools/testing/selftests/bpf/bench.c           |   4 +
>  .../selftests/bpf/benchs/bench_bpf_for.c      | 104 ++++++++++++
>  .../selftests/bpf/benchs/run_bench_bpf_for.sh |  15 ++
>  .../selftests/bpf/progs/bpf_for_bench.c       |  32 ++++
>  tools/testing/selftests/bpf/progs/iters.c     | 145 ++++++++++++++++
>  8 files changed, 472 insertions(+)
>  create mode 100644 tools/testing/selftests/bpf/benchs/bench_bpf_for.c
>  create mode 100755 tools/testing/selftests/bpf/benchs/run_bench_bpf_for.sh
>  create mode 100644 tools/testing/selftests/bpf/progs/bpf_for_bench.c
>
>
> base-commit: a23a71823352e2d792dcaae25f1ebb744acbfc0b