[PATCH bpf-next v5 0/6] bpf: Inline the numeric open-coded iterator kfuncs

[PATCH bpf-next v5 0/6] bpf: Inline the numeric open-coded iterator kfuncs

[Puranjay Mohan]

The bpf_for(i, start, end) macro is BPF's open-coded numeric iterator. It
expands into calls to three kfuncs: bpf_iter_num_new() to set the iterator
up, bpf_iter_num_next() once per iteration, and bpf_iter_num_destroy() to
tear it down. The verifier emits these as ordinary kfunc calls, so a
bpf_for() loop pays function-call overhead on setup, teardown, and -- most
importantly -- on every single iteration via bpf_iter_num_next().

All three kfuncs are tiny and only touch the 8-byte on-stack iterator state
(struct bpf_iter_num_kern { int cur; int end; }). That makes them good
candidates for inlining, the same way several other special kfuncs are
already open-coded in bpf_fixup_kfunc_call(). This series replaces each of
the three calls with an equivalent inline BPF instruction sequence:

  - bpf_iter_num_new(): the end - start range check is done with 32-bit
    arithmetic (start <= end is checked first, so the distance fits in a
    u32) and range-checked against BPF_MAX_LOOPS as unsigned. This avoids
    the cpuv4 sign-extension insns that some JITs do not implement. Returns
    the same -EINVAL / -E2BIG / 0 as the kfunc.

  - bpf_iter_num_next(): the hot path. cur and end are int, so the kfunc's
    s->cur + 1 >= s->end test is an ordinary signed 32-bit compare and the
    inlined code needs no sign extension.

  - bpf_iter_num_destroy(): the stack slot is no longer tracked as iterator
    state once destroy() returns, so nothing needs to be written to it.
    Both the kfunc and the inlined form become a no-op, which just drops the
    call.

The emitted instructions are plain BPF and remain valid for the
interpreter, so interpreter fallback stays correct and no jit_required
marking is needed.

Benchmark (./bench -p 1 --nr_loops 1000000 {bpf-loop,bpf-for}):

    +--------+---------------------+---------------------+---------------------+
    |  arch  |       bpf_loop      | bpf_for non-inlined |   bpf_for inlined   |
    +--------+---------------------+---------------------+---------------------+
    | x86-64 |  4252 M/s (0.24 ns) |  3608 M/s (0.28 ns) |  7417 M/s (0.13 ns) |
    +--------+---------------------+---------------------+---------------------+
    | arm64  |  649 M/s (1.54 ns)  |  548 M/s (1.82 ns)  |  546 M/s (1.83 ns)  |
    +--------+---------------------+---------------------+---------------------+

On x86-64 removing the per-iteration call roughly doubles bpf_for()
throughput. On arm64 it is neutral, and rather than guess why this was
checked with perf: inlining removes ~28% of the executed instructions (the
call) but leaves the cycle count unchanged -- IPC drops from ~4.2 to ~3.0
and backend stalls rise from ~50% to ~66%. The loop is bound by the latency
of the iterator's on-stack counter, not by call overhead:
bpf_iter_num_next() loads s->cur from the stack, increments it and stores it
back each iteration, and the next iteration's load depends on that store.
The removed call instructions were executing in the shadow of that
store->load stall and were never on the critical path.

A small userspace microbenchmark isolates the effect: a same-address
store->load->add round-trip (the shape of the on-stack counter) costs
~6 cycles/iteration on the tested arm64 core but ~1 cycle on x86-64, where
the core collapses the same-address round-trip into a register move (memory
renaming / store-to-load-forwarding elimination). So on x86-64 the loop is
not latency-bound and the per-iteration call dominates -- removing it is the
~2x win -- whereas on arm64 the call fits entirely inside the store->load
stall the loop already has, so adding or removing it changes nothing.

bpf_loop() is shown for reference only; it is a different construct (a
callback invoked per iteration) and this series does not change it. Its
counter lives in a register rather than on the stack, so on arm64 it avoids
the store->load latency above and is faster than bpf_for() there.

Changelog:
v4: https://lore.kernel.org/all/20260729203633.213973-1-puranjay@kernel.org/
Changes in v5:
- Inline the new()/next()/destroy() sequences directly in
  bpf_fixup_kfunc_call() instead of via helper functions (Andrii Nakryiko)
- Trim the code comments; keep the explanation in the bpf_iter.c kfuncs and
  leave only brief comments at the inline sites, and shorten the
  bpf_iter_num_destroy() kfunc to /* no-op */ (Andrii Nakryiko)
- Reword the patch 1 comment so it no longer forward-references the inlined
  bpf_iter_num_next(), which is only added later in the series (bpf-ci)
- Switch the bpf_iter_num_next() comment to the networking multi-line style

v3: https://lore.kernel.org/all/20260722132424.450230-1-puranjay@kernel.org/
Changes in v4:
- Drop the "elide range checks for constant bounds" patch (Andrii Nakryiko)
- bpf_iter_num_new(): range-check the distance against BPF_MAX_LOOPS with an
  unsigned compare (Andrii Nakryiko)
- bpf_iter_num_destroy(): make it a no-op in both the kfunc and the inlined
  form instead of zeroing the iterator state (Andrii Nakryiko)
- New patch: fix the misleading overflow comment in bpf_iter_num_next() and
  drop the redundant (s64) cast; the int wraparound is intentional and
  load-bearing (Andrii Nakryiko)
- bpf_for benchmark: nr_loops is int, matching what bpf_for() expects
  (Andrii Nakryiko)
- Corroborate the arm64/x86 benchmark difference with perf counters and a
  store-to-load-forwarding microbenchmark (Kumar Kartikeya Dwivedi,
  Andrii Nakryiko)

v2: https://lore.kernel.org/bpf/20260717120215.2171057-1-puranjay@kernel.org/
Changes in v3:
- Elide the range checks in bpf_iter_num_new() when start and end are
  constant, marking the registers precise so paths reaching the call with
  different constants are not pruned (Eduard Zingerman)
- Add __xlated selftests pinning the inlined new()/next()/destroy() shapes
  (Eduard Zingerman)
- Use the insn_buf[i++] idiom in the inline helpers (Eduard Zingerman)
- Pick up Acked-by on patch 3

v1: https://lore.kernel.org/all/20260715130430.318421-1-puranjay@kernel.org/
Changes in v2:
- Don't emit sign-extending (movsx) moves; some JITs (e.g. x86-32, mips32,
  sparc64) decode them as a plain move and would miscompile the range check

Puranjay Mohan (6):
  bpf: Correct the overflow check comment in bpf_iter_num_next()
  bpf: Inline bpf_iter_num_new() kfunc
  bpf: Inline bpf_iter_num_next() kfunc
  bpf: Inline bpf_iter_num_destroy() as a no-op
  selftests/bpf: Verify inlined numeric iterator shape with __xlated
  selftests/bpf: Add bpf_for() benchmark

 kernel/bpf/bpf_iter.c                         |  17 ++-
 kernel/bpf/verifier.c                         |  45 ++++++++
 tools/testing/selftests/bpf/Makefile          |   2 +
 tools/testing/selftests/bpf/bench.c           |   4 +
 .../selftests/bpf/benchs/bench_bpf_for.c      | 104 ++++++++++++++++++
 .../selftests/bpf/benchs/run_bench_bpf_for.sh |  15 +++
 .../selftests/bpf/progs/bpf_for_bench.c       |  32 ++++++
 tools/testing/selftests/bpf/progs/iters.c     |  83 ++++++++++++++
 8 files changed, 292 insertions(+), 10 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/benchs/bench_bpf_for.c
 create mode 100755 tools/testing/selftests/bpf/benchs/run_bench_bpf_for.sh
 create mode 100644 tools/testing/selftests/bpf/progs/bpf_for_bench.c


base-commit: 7f333f85f83da9a20a60a1d8bf518c4ae0818e5a
-- 
2.53.0-Meta

[PATCH bpf-next v5 1/6] bpf: Correct the overflow check comment in bpf_iter_num_next()

[Puranjay Mohan]

The comment on the s->cur + 1 >= s->end check claims the (s64) cast is
needed to avoid overflow when s->cur == s->end == INT_MAX. It isn't:
s->cur + 1 is computed in int and wraps before the cast, so the cast
changes nothing (INT_MAX + 1 compares the same either way).

The wraparound is the point. bpf_iter_num_new() sets s->cur = start - 1,
which wraps to INT_MAX for start == INT_MIN, and the wrapping s->cur + 1
brings it back to start. (s64)s->cur + 1 would instead break iterators
starting at INT_MIN.

Drop the cast and reword the comment. No functional change; the wrap is
well-defined under -fno-strict-overflow.

Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
---
 kernel/bpf/bpf_iter.c | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/kernel/bpf/bpf_iter.c b/kernel/bpf/bpf_iter.c
index f5eaeb2493d4a..b235e117e206a 100644
--- a/kernel/bpf/bpf_iter.c
+++ b/kernel/bpf/bpf_iter.c
@@ -802,12 +802,11 @@ __bpf_kfunc int *bpf_iter_num_next(struct bpf_iter_num* it)
 {
 	struct bpf_iter_num_kern *s = (void *)it;
 
-	/* check failed initialization or if we are done (same behavior);
-	 * need to be careful about overflow, so convert to s64 for checks,
-	 * e.g., if s->cur == s->end == INT_MAX, we can't just do
-	 * s->cur + 1 >= s->end
+	/*
+	 * s->cur < s->end while iterating, else s->cur == s->end == 0; the signed
+	 * s->cur + 1 >= s->end holds even when s->cur + 1 wraps (start == INT_MIN).
 	 */
-	if ((s64)(s->cur + 1) >= s->end) {
+	if (s->cur + 1 >= s->end) {
 		s->cur = s->end = 0;
 		return NULL;
 	}
-- 
2.53.0-Meta

[PATCH bpf-next v5 2/6] bpf: Inline bpf_iter_num_new() kfunc

[Puranjay Mohan]

bpf_for() expands to the bpf_iter_num_{new,next,destroy}() kfuncs, which
the verifier emits as regular calls. They are tiny and only touch the
8-byte on-stack iterator state, so open-code them in bpf_fixup_kfunc_call()
like the other special kfuncs there.

Start with bpf_iter_num_new(): R1 points to the iterator, R2/R3 hold
start/end. The inlined sequence mirrors the kfunc and returns the same
-EINVAL / -E2BIG / 0.

start > end is rejected first, so end - start fits in a u32; range-check
it as u32 on both sides ((u32)(end - start) in the kfunc). A movsx-based
check would emit a cpuv4 instruction that some JITs (x86-32, mips32,
sparc64) decode as a plain move and get wrong.

The emitted instructions are plain BPF, so the interpreter path stays
correct and no jit_required marking is needed.

Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
---
 kernel/bpf/bpf_iter.c |  4 ++--
 kernel/bpf/verifier.c | 24 ++++++++++++++++++++++++
 2 files changed, 26 insertions(+), 2 deletions(-)

diff --git a/kernel/bpf/bpf_iter.c b/kernel/bpf/bpf_iter.c
index b235e117e206a..d19f1b2861d22 100644
--- a/kernel/bpf/bpf_iter.c
+++ b/kernel/bpf/bpf_iter.c
@@ -782,8 +782,8 @@ __bpf_kfunc int bpf_iter_num_new(struct bpf_iter_num *it, int start, int end)
 		return -EINVAL;
 	}
 
-	/* avoid overflows, e.g., if start == INT_MIN and end == INT_MAX */
-	if ((s64)end - (s64)start > BPF_MAX_LOOPS) {
+	/* start <= end here, so end - start fits in a u32 without overflow */
+	if ((u32)(end - start) > BPF_MAX_LOOPS) {
 		s->cur = s->end = 0;
 		return -E2BIG;
 	}
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 7439afdc851a7..e0c91f6412228 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -20006,6 +20006,30 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 		insn_buf[4] = BPF_ALU64_REG(BPF_SUB, BPF_REG_0, BPF_REG_1);
 		insn_buf[5] = BPF_ALU64_IMM(BPF_NEG, BPF_REG_0, 0);
 		*cnt = 6;
+	} else if (desc->func_id == special_kfunc_list[KF_bpf_iter_num_new]) {
+		/* inline bpf_iter_num_new(&it, start, end); R1=&it, R2=start, R3=end */
+		int i = 0;
+
+		/* if (start > end) goto einval; */
+		insn_buf[i++] = BPF_JMP32_REG(BPF_JSGT, BPF_REG_2, BPF_REG_3, 8);
+		/* r0 = (u32)end - (u32)start; if (r0 > BPF_MAX_LOOPS) goto e2big; */
+		insn_buf[i++] = BPF_MOV32_REG(BPF_REG_0, BPF_REG_3);
+		insn_buf[i++] = BPF_ALU32_REG(BPF_SUB, BPF_REG_0, BPF_REG_2);
+		insn_buf[i++] = BPF_JMP_IMM(BPF_JGT, BPF_REG_0, BPF_MAX_LOOPS, 8);
+		/* s->cur = start - 1; s->end = end; return 0; */
+		insn_buf[i++] = BPF_ALU32_IMM(BPF_ADD, BPF_REG_2, -1);
+		insn_buf[i++] = BPF_STX_MEM(BPF_W, BPF_REG_1, BPF_REG_2, 0);
+		insn_buf[i++] = BPF_STX_MEM(BPF_W, BPF_REG_1, BPF_REG_3, 4);
+		insn_buf[i++] = BPF_MOV64_IMM(BPF_REG_0, 0);
+		insn_buf[i++] = BPF_JMP_A(5);
+		/* einval: s->cur = s->end = 0; return -EINVAL; */
+		insn_buf[i++] = BPF_ST_MEM(BPF_DW, BPF_REG_1, 0, 0);
+		insn_buf[i++] = BPF_MOV64_IMM(BPF_REG_0, -EINVAL);
+		insn_buf[i++] = BPF_JMP_A(2);
+		/* e2big: s->cur = s->end = 0; return -E2BIG; */
+		insn_buf[i++] = BPF_ST_MEM(BPF_DW, BPF_REG_1, 0, 0);
+		insn_buf[i++] = BPF_MOV64_IMM(BPF_REG_0, -E2BIG);
+		*cnt = i;
 	}
 
 	if (env->insn_aux_data[insn_idx].arg_prog) {
-- 
2.53.0-Meta

[PATCH bpf-next v5 3/6] bpf: Inline bpf_iter_num_next() kfunc

[Puranjay Mohan]

bpf_iter_num_next() runs on every bpf_for() iteration, so inlining it
drops a call from the loop body. R1 points to the iterator; the returned
pointer to s->cur is R1 itself, since s->cur is first.

s->cur and s->end are int, so the kfunc's s->cur + 1 >= s->end is a
signed 32-bit compare and the inlined code needs no sign extension.

Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
---
 kernel/bpf/verifier.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index e0c91f6412228..6bfd715aaa379 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -20030,6 +20030,23 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 		insn_buf[i++] = BPF_ST_MEM(BPF_DW, BPF_REG_1, 0, 0);
 		insn_buf[i++] = BPF_MOV64_IMM(BPF_REG_0, -E2BIG);
 		*cnt = i;
+	} else if (desc->func_id == special_kfunc_list[KF_bpf_iter_num_next]) {
+		/* inline bpf_iter_num_next(&it); R1=&it, returns &s->cur or NULL */
+		int i = 0;
+
+		/* r0 = s->cur + 1; if ((s32)r0 >= s->end) goto done; */
+		insn_buf[i++] = BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1, 0);
+		insn_buf[i++] = BPF_ALU32_IMM(BPF_ADD, BPF_REG_0, 1);
+		insn_buf[i++] = BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, 4);
+		insn_buf[i++] = BPF_JMP32_REG(BPF_JSGE, BPF_REG_0, BPF_REG_2, 3);
+		/* s->cur = r0; return &s->cur; */
+		insn_buf[i++] = BPF_STX_MEM(BPF_W, BPF_REG_1, BPF_REG_0, 0);
+		insn_buf[i++] = BPF_MOV64_REG(BPF_REG_0, BPF_REG_1);
+		insn_buf[i++] = BPF_JMP_A(2);
+		/* done: s->cur = s->end = 0; return NULL; */
+		insn_buf[i++] = BPF_ST_MEM(BPF_DW, BPF_REG_1, 0, 0);
+		insn_buf[i++] = BPF_MOV64_IMM(BPF_REG_0, 0);
+		*cnt = i;
 	}
 
 	if (env->insn_aux_data[insn_idx].arg_prog) {
-- 
2.53.0-Meta

[PATCH bpf-next v5 4/6] bpf: Inline bpf_iter_num_destroy() as a no-op

[Puranjay Mohan]

Once destroy() returns the stack slot is no longer tracked as iterator
state, so zeroing it is dead work. Make the kfunc a no-op and inline the
call to a single BPF_JA 0 (the fixup can't drop the instruction outright,
so emit a nop; the JITs elide it).

Suggested-by: Andrii Nakryiko <andrii@kernel.org>
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
---
 kernel/bpf/bpf_iter.c | 4 +---
 kernel/bpf/verifier.c | 4 ++++
 2 files changed, 5 insertions(+), 3 deletions(-)

diff --git a/kernel/bpf/bpf_iter.c b/kernel/bpf/bpf_iter.c
index d19f1b2861d22..14a5fdfa04214 100644
--- a/kernel/bpf/bpf_iter.c
+++ b/kernel/bpf/bpf_iter.c
@@ -818,9 +818,7 @@ __bpf_kfunc int *bpf_iter_num_next(struct bpf_iter_num* it)
 
 __bpf_kfunc void bpf_iter_num_destroy(struct bpf_iter_num *it)
 {
-	struct bpf_iter_num_kern *s = (void *)it;
-
-	s->cur = s->end = 0;
+	/* no-op */
 }
 
 __bpf_kfunc_end_defs();
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 6bfd715aaa379..64a142870e5b8 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -20047,6 +20047,10 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 		insn_buf[i++] = BPF_ST_MEM(BPF_DW, BPF_REG_1, 0, 0);
 		insn_buf[i++] = BPF_MOV64_IMM(BPF_REG_0, 0);
 		*cnt = i;
+	} else if (desc->func_id == special_kfunc_list[KF_bpf_iter_num_destroy]) {
+		/* bpf_iter_num_destroy() is a no-op; emit a nop to drop the call */
+		insn_buf[0] = BPF_JMP_A(0);
+		*cnt = 1;
 	}
 
 	if (env->insn_aux_data[insn_idx].arg_prog) {
-- 
2.53.0-Meta

[PATCH bpf-next v5 5/6] selftests/bpf: Verify inlined numeric iterator shape with __xlated

[Puranjay Mohan]

Add an __xlated test pinning the inlined bpf_iter_num_{new,next,destroy}()
shapes. The program is __naked, so there is no compiler glue and the whole
sequence is matched instruction for instruction.

Gate it to x86_64 and arm64 (bpf_jit_needs_zext() == false); elsewhere the
verifier interleaves "wN = wN" zero-extensions that would not match. The
inlining is arch independent, so these two are enough.

Suggested-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
---
 tools/testing/selftests/bpf/progs/iters.c | 83 +++++++++++++++++++++++
 1 file changed, 83 insertions(+)

diff --git a/tools/testing/selftests/bpf/progs/iters.c b/tools/testing/selftests/bpf/progs/iters.c
index 0fa70b133d932..62d7df9e80beb 100644
--- a/tools/testing/selftests/bpf/progs/iters.c
+++ b/tools/testing/selftests/bpf/progs/iters.c
@@ -88,6 +88,89 @@ int iter_err_unsafe_asm_loop(const void *ctx)
 	return 0;
 }
 
+/*
+ * Naked function, so there is no compiler-generated glue and the whole inlined program can be
+ * matched. Pinned to arches whose JITs zero-extend 32-bit writes implicitly
+ * (bpf_jit_needs_zext() == false); on arches that need explicit zero-extension the verifier
+ * interleaves "wN = wN" insns and the fixed shape below would not match. The inlining itself is
+ * arch independent, so checking it on these arches is sufficient.
+ *
+ * bpf_iter_num_new() emits the full range check (distance computation and both the -EINVAL and
+ * -E2BIG error paths); bpf_iter_num_next() and bpf_iter_num_destroy() are inlined too.
+ */
+SEC("raw_tp")
+__arch_x86_64
+__arch_arm64
+__success
+__xlated("r6 = r10")
+__xlated("r6 += -8")
+__xlated("call unknown")
+__xlated("r3 = r0")
+__xlated("r3 &= 65535")
+__xlated("r1 = r6")
+__xlated("r2 = 0")
+/* bpf_iter_num_new(&it, 0, <non-const>) with the range check kept */
+__xlated("if w2 s> w3 goto pc+8")
+__xlated("w0 = w3")
+__xlated("w0 -= w2")
+__xlated("if r0 > 0x800000 goto pc+8")
+__xlated("w2 += -1")
+__xlated("*(u32 *)(r1 +0) = r2")
+__xlated("*(u32 *)(r1 +4) = r3")
+__xlated("r0 = 0")
+__xlated("goto pc+5")
+__xlated("*(u64 *)(r1 +0) = 0")
+__xlated("r0 = -22")
+__xlated("goto pc+2")
+__xlated("*(u64 *)(r1 +0) = 0")
+__xlated("r0 = -7")
+__xlated("r1 = r6")
+/* bpf_iter_num_next(&it) */
+__xlated("r0 = *(u32 *)(r1 +0)")
+__xlated("w0 += 1")
+__xlated("r2 = *(u32 *)(r1 +4)")
+__xlated("if w0 s>= w2 goto pc+3")
+__xlated("*(u32 *)(r1 +0) = r0")
+__xlated("r0 = r1")
+__xlated("goto pc+2")
+__xlated("*(u64 *)(r1 +0) = 0")
+__xlated("r0 = 0")
+__xlated("if r0 != 0x0 goto pc-11")
+__xlated("r1 = r6")
+/* bpf_iter_num_destroy(&it) is inlined to a nop */
+__xlated("goto pc+0")
+__xlated("r0 = 0")
+__xlated("exit")
+int __naked iter_num_new_inlined(void)
+{
+	asm volatile (
+		/* r6 points to struct bpf_iter_num on the stack */
+		"r6 = r10;"
+		"r6 += -8;"
+		/* non-constant end so the range checks are kept */
+		"call %[bpf_get_prandom_u32];"
+		"r3 = r0;"
+		"r3 &= 0xffff;"
+		"r1 = r6;"
+		"r2 = 0;"
+		"call %[bpf_iter_num_new];"
+	"1:"
+		"r1 = r6;"
+		"call %[bpf_iter_num_next];"
+		"if r0 != 0 goto 1b;"
+		"r1 = r6;"
+		"call %[bpf_iter_num_destroy];"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm(bpf_get_prandom_u32),
+		  __imm(bpf_iter_num_new),
+		  __imm(bpf_iter_num_next),
+		  __imm(bpf_iter_num_destroy)
+		: __clobber_common, "r6"
+	);
+}
+
 SEC("raw_tp")
 __success
 int iter_while_loop(const void *ctx)
-- 
2.53.0-Meta

[PATCH bpf-next v5 6/6] selftests/bpf: Add bpf_for() benchmark

[Puranjay Mohan]

Add a bpf_for() benchmark modelled on bench_bpf_loop so the per-iteration
iterator cost can be measured and compared against bpf_loop. It runs an
empty bpf_for(i, 0, nr_loops) loop 1000 times per trigger and accounts
nr_loops hits per outer iteration:

  $ ./bench -p 1 --nr_loops 1000 bpf-for

Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
---
 tools/testing/selftests/bpf/Makefile          |   2 +
 tools/testing/selftests/bpf/bench.c           |   4 +
 .../selftests/bpf/benchs/bench_bpf_for.c      | 104 ++++++++++++++++++
 .../selftests/bpf/benchs/run_bench_bpf_for.sh |  15 +++
 .../selftests/bpf/progs/bpf_for_bench.c       |  32 ++++++
 5 files changed, 157 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/benchs/bench_bpf_for.c
 create mode 100755 tools/testing/selftests/bpf/benchs/run_bench_bpf_for.sh
 create mode 100644 tools/testing/selftests/bpf/progs/bpf_for_bench.c

diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile
index 55d394438705a..3174c2f18e715 100644
--- a/tools/testing/selftests/bpf/Makefile
+++ b/tools/testing/selftests/bpf/Makefile
@@ -965,6 +965,7 @@ $(OUTPUT)/bench_ringbufs.o: $(OUTPUT)/ringbuf_bench.skel.h \
 			    $(OUTPUT)/perfbuf_bench.skel.h
 $(OUTPUT)/bench_bloom_filter_map.o: $(OUTPUT)/bloom_filter_bench.skel.h
 $(OUTPUT)/bench_bpf_loop.o: $(OUTPUT)/bpf_loop_bench.skel.h
+$(OUTPUT)/bench_bpf_for.o: $(OUTPUT)/bpf_for_bench.skel.h
 $(OUTPUT)/bench_strncmp.o: $(OUTPUT)/strncmp_bench.skel.h
 $(OUTPUT)/bench_bpf_hashmap_full_update.o: $(OUTPUT)/bpf_hashmap_full_update_bench.skel.h
 $(OUTPUT)/bench_local_storage.o: $(OUTPUT)/local_storage_bench.skel.h
@@ -990,6 +991,7 @@ $(OUTPUT)/bench: $(OUTPUT)/bench.o \
 		 $(OUTPUT)/bench_ringbufs.o \
 		 $(OUTPUT)/bench_bloom_filter_map.o \
 		 $(OUTPUT)/bench_bpf_loop.o \
+		 $(OUTPUT)/bench_bpf_for.o \
 		 $(OUTPUT)/bench_strncmp.o \
 		 $(OUTPUT)/bench_bpf_hashmap_full_update.o \
 		 $(OUTPUT)/bench_local_storage.o \
diff --git a/tools/testing/selftests/bpf/bench.c b/tools/testing/selftests/bpf/bench.c
index 3d9d2cd7764bd..b86b73456d3ca 100644
--- a/tools/testing/selftests/bpf/bench.c
+++ b/tools/testing/selftests/bpf/bench.c
@@ -276,6 +276,7 @@ static const struct argp_option opts[] = {
 extern struct argp bench_ringbufs_argp;
 extern struct argp bench_bloom_map_argp;
 extern struct argp bench_bpf_loop_argp;
+extern struct argp bench_bpf_for_argp;
 extern struct argp bench_local_storage_argp;
 extern struct argp bench_local_storage_rcu_tasks_trace_argp;
 extern struct argp bench_strncmp_argp;
@@ -292,6 +293,7 @@ static const struct argp_child bench_parsers[] = {
 	{ &bench_ringbufs_argp, 0, "Ring buffers benchmark", 0 },
 	{ &bench_bloom_map_argp, 0, "Bloom filter map benchmark", 0 },
 	{ &bench_bpf_loop_argp, 0, "bpf_loop helper benchmark", 0 },
+	{ &bench_bpf_for_argp, 0, "bpf_for loop benchmark", 0 },
 	{ &bench_local_storage_argp, 0, "local_storage benchmark", 0 },
 	{ &bench_strncmp_argp, 0, "bpf_strncmp helper benchmark", 0 },
 	{ &bench_local_storage_rcu_tasks_trace_argp, 0,
@@ -557,6 +559,7 @@ extern const struct bench bench_bloom_false_positive;
 extern const struct bench bench_hashmap_without_bloom;
 extern const struct bench bench_hashmap_with_bloom;
 extern const struct bench bench_bpf_loop;
+extern const struct bench bench_bpf_for;
 extern const struct bench bench_strncmp_no_helper;
 extern const struct bench bench_strncmp_helper;
 extern const struct bench bench_bpf_hashmap_full_update;
@@ -640,6 +643,7 @@ static const struct bench *benchs[] = {
 	&bench_hashmap_without_bloom,
 	&bench_hashmap_with_bloom,
 	&bench_bpf_loop,
+	&bench_bpf_for,
 	&bench_strncmp_no_helper,
 	&bench_strncmp_helper,
 	&bench_bpf_hashmap_full_update,
diff --git a/tools/testing/selftests/bpf/benchs/bench_bpf_for.c b/tools/testing/selftests/bpf/benchs/bench_bpf_for.c
new file mode 100644
index 0000000000000..730c51ad2dec3
--- /dev/null
+++ b/tools/testing/selftests/bpf/benchs/bench_bpf_for.c
@@ -0,0 +1,104 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+#include <argp.h>
+#include "bench.h"
+#include "bpf_for_bench.skel.h"
+
+/* BPF triggering benchmarks */
+static struct ctx {
+	struct bpf_for_bench *skel;
+} ctx;
+
+static struct {
+	__u32 nr_loops;
+} args = {
+	/*
+	 * Default to a large loop count so the per-iteration bpf_iter_num_next() cost dominates
+	 * the one-time bpf_iter_num_new()/destroy() setup and teardown.
+	 */
+	.nr_loops = 1000,
+};
+
+enum {
+	ARG_NR_LOOPS = 4000,
+};
+
+static const struct argp_option opts[] = {
+	{ "nr_loops", ARG_NR_LOOPS, "nr_loops", 0,
+		"Set number of iterations for the bpf_for() loop"},
+	{},
+};
+
+static error_t parse_arg(int key, char *arg, struct argp_state *state)
+{
+	switch (key) {
+	case ARG_NR_LOOPS:
+		args.nr_loops = strtol(arg, NULL, 10);
+		break;
+	default:
+		return ARGP_ERR_UNKNOWN;
+	}
+
+	return 0;
+}
+
+/* exported into benchmark runner */
+const struct argp bench_bpf_for_argp = {
+	.options = opts,
+	.parser = parse_arg,
+};
+
+static void validate(void)
+{
+	if (env.consumer_cnt != 0) {
+		fprintf(stderr, "benchmark doesn't support consumer!\n");
+		exit(1);
+	}
+}
+
+static void *producer(void *input)
+{
+	while (true)
+		/* trigger the bpf program */
+		syscall(__NR_getpgid);
+
+	return NULL;
+}
+
+static void measure(struct bench_res *res)
+{
+	res->hits = atomic_swap(&ctx.skel->bss->hits, 0);
+}
+
+static void setup(void)
+{
+	struct bpf_link *link;
+
+	setup_libbpf();
+
+	ctx.skel = bpf_for_bench__open_and_load();
+	if (!ctx.skel) {
+		fprintf(stderr, "failed to open skeleton\n");
+		exit(1);
+	}
+
+	link = bpf_program__attach(ctx.skel->progs.benchmark);
+	if (!link) {
+		fprintf(stderr, "failed to attach program!\n");
+		exit(1);
+	}
+
+	ctx.skel->bss->nr_loops = args.nr_loops;
+}
+
+const struct bench bench_bpf_for = {
+	.name = "bpf-for",
+	.argp = &bench_bpf_for_argp,
+	.validate = validate,
+	.setup = setup,
+	.producer_thread = producer,
+	.measure = measure,
+	.report_progress = ops_report_progress,
+	.report_final = ops_report_final,
+};
diff --git a/tools/testing/selftests/bpf/benchs/run_bench_bpf_for.sh b/tools/testing/selftests/bpf/benchs/run_bench_bpf_for.sh
new file mode 100755
index 0000000000000..7da6453920dab
--- /dev/null
+++ b/tools/testing/selftests/bpf/benchs/run_bench_bpf_for.sh
@@ -0,0 +1,15 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+
+source ./benchs/run_common.sh
+
+set -eufo pipefail
+
+for t in 1 4 8 12 16; do
+for i in 10 100 500 1000 5000 10000 50000 100000 500000 1000000; do
+subtitle "nr_loops: $i, nr_threads: $t"
+	summarize_ops "bpf_for: " \
+	    "$($RUN_BENCH -p $t --nr_loops $i bpf-for)"
+	printf "\n"
+done
+done
diff --git a/tools/testing/selftests/bpf/progs/bpf_for_bench.c b/tools/testing/selftests/bpf/progs/bpf_for_bench.c
new file mode 100644
index 0000000000000..f9c723051fc74
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/bpf_for_bench.c
@@ -0,0 +1,32 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+#include "vmlinux.h"
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+char _license[] SEC("license") = "GPL";
+
+int nr_loops;
+long hits;
+
+static int outer_loop(__u32 index, void *data)
+{
+	int i;
+
+	/*
+	 * Empty body: the work being measured is the open-coded numeric iterator itself
+	 * (bpf_iter_num_new/next/destroy behind bpf_for()).
+	 */
+	bpf_for(i, 0, nr_loops)
+		;
+	__sync_add_and_fetch(&hits, nr_loops);
+	return 0;
+}
+
+SEC("fentry/" SYS_PREFIX "sys_getpgid")
+int benchmark(void *ctx)
+{
+	bpf_loop(1000, outer_loop, NULL, 0);
+	return 0;
+}
-- 
2.53.0-Meta

Re: [PATCH bpf-next v5 1/6] bpf: Correct the overflow check comment in bpf_iter_num_next()

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 2406 bytes --]

> diff --git a/kernel/bpf/bpf_iter.c b/kernel/bpf/bpf_iter.c
> index f5eaeb2493d4a..b235e117e206a 100644
> --- a/kernel/bpf/bpf_iter.c
> +++ b/kernel/bpf/bpf_iter.c
> @@ -802,12 +802,11 @@ __bpf_kfunc int *bpf_iter_num_next(struct bpf_iter_num* it)
>  {
>  	struct bpf_iter_num_kern *s = (void *)it;
>
> -	/* check failed initialization or if we are done (same behavior);
> -	 * need to be careful about overflow, so convert to s64 for checks,
> -	 * e.g., if s->cur == s->end == INT_MAX, we can't just do
> -	 * s->cur + 1 >= s->end
> +	/*
> +	 * s->cur < s->end while iterating, else s->cur == s->end == 0; the signed
> +	 * s->cur + 1 >= s->end holds even when s->cur + 1 wraps (start == INT_MIN).
                                  ^^^^^
>  	 */
> -	if ((s64)(s->cur + 1) >= s->end) {
> +	if (s->cur + 1 >= s->end) {
>  		s->cur = s->end = 0;
>  		return NULL;
>  	}

The replacement comment says "s->cur + 1 >= s->end holds even when
s->cur + 1 wraps (start == INT_MIN)". Doesn't the inequality evaluate to
false in the wrap case, not true?

Looking at the code: bpf_iter_num_new() sets s->cur = start - 1, so when
start == INT_MIN, s->cur wraps to INT_MAX. On the first call to
bpf_iter_num_next(), s->cur + 1 wraps back to INT_MIN. For a non-empty
range like the selftest bpf_for(i, INT_MIN, INT_MIN + 2) at
tools/testing/selftests/bpf/progs/iters_num.c:101, the test becomes
INT_MIN >= INT_MIN + 2, which is false, so the function returns the first
element.

The commit message describes this correctly ("the wrapping s->cur + 1
brings it back to start"), but "X >= Y holds" reads as "the inequality is
true," when what seems to be meant is that the comparison mechanism remains
correct under wraparound.

Also, the first clause says "s->cur < s->end while iterating, else
s->cur == s->end == 0". This doesn't cover the post-bpf_iter_num_new(),
pre-first-next() state for start == INT_MIN, where s->cur == INT_MAX and
s->end == INT_MIN + 2 (neither s->cur < s->end nor s->cur == s->end == 0).

Since the sole purpose of this patch is to make the comment accurate, could
the wording be adjusted to avoid asserting a condition that is false in the
cited case?


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/30915992491

Re: [PATCH bpf-next v5 1/6] bpf: Correct the overflow check comment in bpf_iter_num_next()

[Puranjay Mohan]

On Tue, Aug 4, 2026 at 3:47 PM <bot+bpf-ci@kernel.org> wrote:
>
> > diff --git a/kernel/bpf/bpf_iter.c b/kernel/bpf/bpf_iter.c
> > index f5eaeb2493d4a..b235e117e206a 100644
> > --- a/kernel/bpf/bpf_iter.c
> > +++ b/kernel/bpf/bpf_iter.c
> > @@ -802,12 +802,11 @@ __bpf_kfunc int *bpf_iter_num_next(struct bpf_iter_num* it)
> >  {
> >       struct bpf_iter_num_kern *s = (void *)it;
> >
> > -     /* check failed initialization or if we are done (same behavior);
> > -      * need to be careful about overflow, so convert to s64 for checks,
> > -      * e.g., if s->cur == s->end == INT_MAX, we can't just do
> > -      * s->cur + 1 >= s->end
> > +     /*
> > +      * s->cur < s->end while iterating, else s->cur == s->end == 0; the signed
> > +      * s->cur + 1 >= s->end holds even when s->cur + 1 wraps (start == INT_MIN).
>                                   ^^^^^
> >        */
> > -     if ((s64)(s->cur + 1) >= s->end) {
> > +     if (s->cur + 1 >= s->end) {
> >               s->cur = s->end = 0;
> >               return NULL;
> >       }
>
> The replacement comment says "s->cur + 1 >= s->end holds even when
> s->cur + 1 wraps (start == INT_MIN)". Doesn't the inequality evaluate to
> false in the wrap case, not true?
>
> Looking at the code: bpf_iter_num_new() sets s->cur = start - 1, so when
> start == INT_MIN, s->cur wraps to INT_MAX. On the first call to
> bpf_iter_num_next(), s->cur + 1 wraps back to INT_MIN. For a non-empty
> range like the selftest bpf_for(i, INT_MIN, INT_MIN + 2) at
> tools/testing/selftests/bpf/progs/iters_num.c:101, the test becomes
> INT_MIN >= INT_MIN + 2, which is false, so the function returns the first
> element.
>
> The commit message describes this correctly ("the wrapping s->cur + 1
> brings it back to start"), but "X >= Y holds" reads as "the inequality is
> true," when what seems to be meant is that the comparison mechanism remains
> correct under wraparound.
>
> Also, the first clause says "s->cur < s->end while iterating, else
> s->cur == s->end == 0". This doesn't cover the post-bpf_iter_num_new(),
> pre-first-next() state for start == INT_MIN, where s->cur == INT_MAX and
> s->end == INT_MIN + 2 (neither s->cur < s->end nor s->cur == s->end == 0).
>
> Since the sole purpose of this patch is to make the comment accurate, could
> the wording be adjusted to avoid asserting a condition that is false in the
> cited case?

Andrii do you have a preference for the comment here? I will copy
whatever you say verbatim to the next version.

Thanks,
Puranjay

Re: [PATCH bpf-next v5 0/6] bpf: Inline the numeric open-coded iterator kfuncs

[patchwork-bot+netdevbpf]

Hello:

This series was applied to bpf/bpf-next.git (master)
by Andrii Nakryiko <andrii@kernel.org>:

On Tue,  4 Aug 2026 06:45:52 -0700 you wrote:
> The bpf_for(i, start, end) macro is BPF's open-coded numeric iterator. It
> expands into calls to three kfuncs: bpf_iter_num_new() to set the iterator
> up, bpf_iter_num_next() once per iteration, and bpf_iter_num_destroy() to
> tear it down. The verifier emits these as ordinary kfunc calls, so a
> bpf_for() loop pays function-call overhead on setup, teardown, and -- most
> importantly -- on every single iteration via bpf_iter_num_next().
> 
> [...]

Here is the summary with links:
  - [bpf-next,v5,1/6] bpf: Correct the overflow check comment in bpf_iter_num_next()
    https://git.kernel.org/bpf/bpf-next/c/8efd87051c3a
  - [bpf-next,v5,2/6] bpf: Inline bpf_iter_num_new() kfunc
    https://git.kernel.org/bpf/bpf-next/c/f8f2b567d560
  - [bpf-next,v5,3/6] bpf: Inline bpf_iter_num_next() kfunc
    https://git.kernel.org/bpf/bpf-next/c/e93347704878
  - [bpf-next,v5,4/6] bpf: Inline bpf_iter_num_destroy() as a no-op
    https://git.kernel.org/bpf/bpf-next/c/39f047682fe3
  - [bpf-next,v5,5/6] selftests/bpf: Verify inlined numeric iterator shape with __xlated
    https://git.kernel.org/bpf/bpf-next/c/b829bc167705
  - [bpf-next,v5,6/6] selftests/bpf: Add bpf_for() benchmark
    https://git.kernel.org/bpf/bpf-next/c/5a1de41147b0

You are awesome, thank you!
-- 
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html