Re: [PATCH bpf-next v3 0/6] bpf: Inline the numeric open-coded iterator kfuncs

["Kumar Kartikeya Dwivedi" <memxor@gmail.com>]

On Tue Jul 28, 2026 at 5:59 PM CEST, Puranjay Mohan wrote:
> On Tue, Jul 28, 2026 at 4:56 PM Andrii Nakryiko
> <andrii.nakryiko@gmail.com> wrote:
>>
>> On Wed, Jul 22, 2026 at 6:24 AM Puranjay Mohan <puranjay@kernel.org> wrote:
>> >
>> > The bpf_for(i, start, end) macro is BPF's open-coded numeric iterator. It
>> > expands into calls to three kfuncs: bpf_iter_num_new() to set the iterator
>> > up, bpf_iter_num_next() once per iteration, and bpf_iter_num_destroy() to
>> > tear it down. The verifier emits these as ordinary kfunc calls, so a
>> > bpf_for() loop pays function-call overhead on setup, teardown, and — most
>> > importantly — on every single iteration via bpf_iter_num_next().
>> >
>> > All three kfuncs are tiny and only touch the 8-byte on-stack iterator state
>> > (struct bpf_iter_num_kern { int cur; int end; }). That makes them good
>> > candidates for inlining, the same way several other special kfuncs are
>> > already open-coded in bpf_fixup_kfunc_call(). This series replaces each of
>> > the three calls with an equivalent inline BPF instruction sequence:
>> >
>> >   - bpf_iter_num_new(): the (s64)end - (s64)start overflow check is done
>> >     with 32-bit arithmetic (start <= end is checked first, so the range
>> >     fits in a u32), avoiding cpuv4 sign-extension insns that some JITs do
>> >     not implement. Returns the same -EINVAL / -E2BIG / 0 as the kfunc.
>> >
>> >   - bpf_iter_num_next(): the hot path. Since cur and end are int, the
>> >     kfunc's (s64)(s->cur + 1) >= s->end test reduces to a signed 32-bit
>> >     comparison of (s->cur + 1) against s->end, so the inlined code uses a
>> >     32-bit compare with no sign extension.
>> >
>> >   - bpf_iter_num_destroy(): a single 8-byte store zeroing the state.
>>
>> tbh, I don't think we need to do *anything* in bpf_iter_num_destroy(),
>> not sure why I added that zeroing out there. that stack slot will not
>> be treated as iterator state after destroy() is called, so what is
>> stored there doesn't matter, we can simplify (unless I'm missing
>> something non-obvious, but I can't see it)
>>
>> >
>> > bpf_for() is frequently used with constant bounds; in that case the range
>> > checks in bpf_iter_num_new() are decidable at verification time, so a
>> > follow-up patch elides them and emits only the iterator init (or the error
>> > result). The inlined shapes are pinned by new __xlated selftests, and a
>> > bench_bpf_for benchmark (modeled on the existing bpf_loop benchmark) runs a
>> > bpf_for() loop with an empty body to measure the per-iteration cost.
>> >
>> > The emitted instructions are plain BPF and remain valid for the
>> > interpreter, so interpreter fallback stays correct and no jit_required
>> > marking is needed.
>> >
>> > Benchmark (./bench -p 1 --nr_loops 1000000 {bpf-loop,bpf-for}):
>> >
>> >     +--------+---------------------+---------------------+---------------------+
>> >     |  arch  |       bpf_loop      | bpf_for non-inlined |   bpf_for inlined   |
>> >     +--------+---------------------+---------------------+---------------------+
>> >     | x86-64 | 2946 M/s (0.339 ns) | 4281 M/s (0.234 ns) | 8981 M/s (0.111 ns) |
>> >     +--------+---------------------+---------------------+---------------------+
>> >     | arm64  |  618 M/s (1.619 ns) |  543 M/s (1.843 ns) |  538 M/s (1.858 ns) |
>> >     +--------+---------------------+---------------------+---------------------+
>> >
>> > On x86-64, removing the per-iteration call to bpf_iter_num_next() roughly
>> > doubles bpf_for() throughput. On arm64 it is neutral: the loop is bound by
>> > the load/store dependency chain on the on-stack iterator state rather than
>> > by call overhead, so inlining neither helps nor hurts there. It still
>> > removes the calls.
>>
>> exactly, and those removed calls should be noticeable in such
>> microbenchmark... also, bpf_loop being faster than bpf_for() is a big
>> surprise. let's dive deeper into this, this makes no sense
>
> Yes, I will do more perf testing before posting the next version.
>
>> > bpf_loop() is shown for reference only; it is a different construct (a
>> > callback invoked per iteration), so the comparison is structural rather
>> > than a measure of the inlining: on x86-64 bpf_for() is faster than
>> > bpf_loop() even before inlining, while on arm64 bpf_loop() is faster
>> > because bpf_for()'s per-iteration cost is dominated by the stack round-trip
>> > through the iterator state.
>>
>> hm, again, I'm not sure I'm buying this. bpf_loop() does indirect
>> function call, stack access can't be more expensive than that, can
>> it?..
>
> No, the above explanation given by me is not accurate, it was my
> theory but it didn't hold up when I looked into perf reports. So, I
> need to dive deeper into this.

It might also be worth comparing cases where verifier does inline and does not
inline bpf_loop, the latter can often be the common case for some programs, and
may help appreciate the improvement more.