Re: [PATCH bpf-next 0/4] bpf: Inline the numeric open-coded iterator kfuncs

[Leon Hwang <leon.hwang@linux.dev>]

On 2026/7/15 21:04, Puranjay Mohan wrote:
> The bpf_for(i, start, end) macro is BPF's open-coded numeric iterator. It
> expands into calls to three kfuncs: bpf_iter_num_new() to set the iterator
> up, bpf_iter_num_next() once per iteration, and bpf_iter_num_destroy() to
> tear it down. The verifier emits these as ordinary kfunc calls, so a
> bpf_for() loop pays function-call overhead on setup, teardown, and — most
> importantly — on every single iteration via bpf_iter_num_next().
> 
> All three kfuncs are tiny and only touch the 8-byte on-stack iterator state
> (struct bpf_iter_num_kern { int cur; int end; }). That makes them good
> candidates for inlining, the same way several other special kfuncs are
> already open-coded in bpf_fixup_kfunc_call(). This series replaces each of
> the three calls with an equivalent inline BPF instruction sequence:
> 
>   - bpf_iter_num_new(): keeps the (s64)end - (s64)start overflow check
>     (emitted with sign-extending moves) and returns the same
>     -EINVAL / -E2BIG / 0 results as the kfunc.
> 
>   - bpf_iter_num_next(): the hot path. Since cur and end are int, the
>     kfunc's (s64)(s->cur + 1) >= s->end test reduces to a signed 32-bit
>     comparison of (s->cur + 1) against s->end, so the inlined code uses a
>     32-bit compare with no sign extension.
> 
>   - bpf_iter_num_destroy(): a single 8-byte store zeroing the state.
> 
> The emitted instructions are plain BPF and remain valid for the
> interpreter, so interpreter fallback stays correct and no jit_required
> marking is needed.
> 
> Patch 4 adds a bench_bpf_for benchmark, modeled on the existing bpf_loop
> benchmark, that runs a bpf_for() loop with an empty body so the
> per-iteration iterator cost can be measured directly.
> 
> Results (./bench -p 1 --nr_loops 1000 bpf-for):
> 
>   +--------+----------------------+----------------------+--------------+
>   |  arch  |       inlined        |     kfunc calls      |   result     |
>   +--------+----------------------+----------------------+--------------+
>   | x86-64 | 12884 M/s (0.078 ns) | 5812 M/s (0.172 ns)  | ~2.2x faster |
>   +--------+----------------------+----------------------+--------------+
>   | arm64  | 546.1 M/s (1.831 ns) | 545.8 M/s (1.832 ns) | neutral      |
>   +--------+----------------------+----------------------+--------------+
> 


The bench results are excellent on x86-64.

However, after looking through patch #1 and patch #2, hmm, they seem
quite complex.

Alexei suggested that the kfuncs could be inlined by the kernel disasm
approach [1]. With that approach, it should be possible to inline many
kfuncs and helpers on both x86-64 and arm64.

I think we should try hard to implement the disasm approach.

[1]
https://lore.kernel.org/bpf/CAADnVQLNmQGKf5S5ZNwHYzScYBhnWFmnzLg=5Xxy4SgYKE3EfQ@mail.gmail.com/

Thanks,
Leon

> On x86-64, removing the per-iteration call to bpf_iter_num_next() roughly
> doubles throughput. On arm64 the numbers are unchanged: the loop is bound
> by the load/store dependency chain on the on-stack iterator state rather
> than by call overhead, so inlining neither helps nor hurts there. It still
> removes the calls.
> 
> [...]