[PATCH bpf-next v2 0/4] bpf: Inline the numeric open-coded iterator kfuncs

[PATCH bpf-next v2 0/4] bpf: Inline the numeric open-coded iterator kfuncs

[Puranjay Mohan]

The bpf_for(i, start, end) macro is BPF's open-coded numeric iterator. It
expands into calls to three kfuncs: bpf_iter_num_new() to set the iterator
up, bpf_iter_num_next() once per iteration, and bpf_iter_num_destroy() to
tear it down. The verifier emits these as ordinary kfunc calls, so a
bpf_for() loop pays function-call overhead on setup, teardown, and — most
importantly — on every single iteration via bpf_iter_num_next().

All three kfuncs are tiny and only touch the 8-byte on-stack iterator state
(struct bpf_iter_num_kern { int cur; int end; }). That makes them good
candidates for inlining, the same way several other special kfuncs are
already open-coded in bpf_fixup_kfunc_call(). This series replaces each of
the three calls with an equivalent inline BPF instruction sequence:

  - bpf_iter_num_new(): keeps the (s64)end - (s64)start overflow check
    (emitted with sign-extending moves) and returns the same
    -EINVAL / -E2BIG / 0 results as the kfunc.

  - bpf_iter_num_next(): the hot path. Since cur and end are int, the
    kfunc's (s64)(s->cur + 1) >= s->end test reduces to a signed 32-bit
    comparison of (s->cur + 1) against s->end, so the inlined code uses a
    32-bit compare with no sign extension.

  - bpf_iter_num_destroy(): a single 8-byte store zeroing the state.

The emitted instructions are plain BPF and remain valid for the
interpreter, so interpreter fallback stays correct and no jit_required
marking is needed.

Patch 4 adds a bench_bpf_for benchmark, modeled on the existing bpf_loop
benchmark, that runs a bpf_for() loop with an empty body so the
per-iteration iterator cost can be measured directly.

Results (./bench -p 1 --nr_loops 1000 bpf-for):

  +--------+----------------------+----------------------+--------------+
  |  arch  |       inlined        |     kfunc calls      |   result     |
  +--------+----------------------+----------------------+--------------+
  | x86-64 | 12884 M/s (0.078 ns) | 5812 M/s (0.172 ns)  | ~2.2x faster |
  +--------+----------------------+----------------------+--------------+
  | arm64  | 546.1 M/s (1.831 ns) | 545.8 M/s (1.832 ns) | neutral      |
  +--------+----------------------+----------------------+--------------+

On x86-64, removing the per-iteration call to bpf_iter_num_next() roughly
doubles throughput. On arm64 the numbers are unchanged: the loop is bound
by the load/store dependency chain on the on-stack iterator state rather
than by call overhead, so inlining neither helps nor hurts there. It still
removes the calls.

Changelog:
v1: https://lore.kernel.org/all/20260715130430.318421-1-puranjay@kernel.org/
Changes in v2:
- Don't use sign-extending moves as some JITs don't support them (AI)

Puranjay Mohan (4):
  bpf: Inline bpf_iter_num_new() kfunc
  bpf: Inline bpf_iter_num_next() kfunc
  bpf: Inline bpf_iter_num_destroy() kfunc
  selftests/bpf: Add bpf_for() benchmark

 kernel/bpf/verifier.c                         |  81 ++++++++++++++
 tools/testing/selftests/bpf/Makefile          |   2 +
 tools/testing/selftests/bpf/bench.c           |   4 +
 .../selftests/bpf/benchs/bench_bpf_for.c      | 105 ++++++++++++++++++
 .../selftests/bpf/benchs/run_bench_bpf_for.sh |  15 +++
 .../selftests/bpf/progs/bpf_for_bench.c       |  32 ++++++
 6 files changed, 239 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/benchs/bench_bpf_for.c
 create mode 100755 tools/testing/selftests/bpf/benchs/run_bench_bpf_for.sh
 create mode 100644 tools/testing/selftests/bpf/progs/bpf_for_bench.c


base-commit: 1d91ea01185656ac3ee63c5f9f6f8bde3c746b3d
-- 
2.53.0-Meta

[PATCH bpf-next v2 1/4] bpf: Inline bpf_iter_num_new() kfunc

[Puranjay Mohan]

The numeric iterator kfuncs bpf_iter_num_{new,next,destroy}() back the
open-coded iterator macro bpf_for() and are emitted as regular kfunc
calls by the verifier. bpf_iter_num_new() is small and only touches the
on-stack iterator state, so the verifier can open-code it and avoid the
call overhead of setting up an iterator.

Inline it in bpf_fixup_kfunc_call() by replacing the call with an
equivalent instruction sequence. R1 holds the pointer to the on-stack
bpf_iter_num, while R2 and R3 hold the start and end arguments. The
arithmetic and the error paths mirror the kfunc exactly, so a program
that inspects the return value keeps observing the same -EINVAL /
-E2BIG / 0 results.

The kfunc guards against overflow with a (s64)end - (s64)start range
check. The start > end case is rejected first, so start <= end at that
point and the range end - start fits in a u32. The inlined code computes
it with a 32-bit subtraction that zero-extends into a 64-bit register
and compares the result against BPF_MAX_LOOPS. Sign-extending the
operands with movsx instead would emit a cpuv4 instruction after
verification; JITs that do not implement movsx (e.g. x86-32, mips32,
sparc64) decode it as a plain move and would miscompile the check.

The emitted instructions are plain BPF and are handled by the
interpreter, so interpreter fallback stays correct and no jit_required
marking is needed.

Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
---
 kernel/bpf/verifier.c | 38 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)

diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index a78cdabf85607..9b9748b2a7cf5 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -19721,6 +19721,42 @@ static void __fixup_collection_insert_kfunc(struct bpf_insn_aux_data *insn_aux,
 	*cnt = 4;
 }
 
+/*
+ * Inline bpf_iter_num_new(). R1 holds the pointer to the iterator, R2 and R3
+ * hold the (int) start and end arguments. Keep in sync with the kfunc in
+ * kernel/bpf/bpf_iter.c.
+ */
+static int inline_bpf_iter_num_new(struct bpf_insn *insn_buf)
+{
+	/* if (start > end) goto einval; */
+	insn_buf[0] = BPF_JMP32_REG(BPF_JSGT, BPF_REG_2, BPF_REG_3, 8);
+	/*
+	 * start <= end here, so the range end - start fits in a u32; compute it
+	 * as a 32-bit subtraction that zero-extends into r0.
+	 */
+	insn_buf[1] = BPF_MOV32_REG(BPF_REG_0, BPF_REG_3);
+	insn_buf[2] = BPF_ALU32_REG(BPF_SUB, BPF_REG_0, BPF_REG_2);
+	/* if (r0 > BPF_MAX_LOOPS) goto e2big; */
+	insn_buf[3] = BPF_JMP_IMM(BPF_JSGT, BPF_REG_0, BPF_MAX_LOOPS, 8);
+	/* s->cur = start - 1; */
+	insn_buf[4] = BPF_ALU32_IMM(BPF_ADD, BPF_REG_2, -1);
+	insn_buf[5] = BPF_STX_MEM(BPF_W, BPF_REG_1, BPF_REG_2, 0);
+	/* s->end = end; */
+	insn_buf[6] = BPF_STX_MEM(BPF_W, BPF_REG_1, BPF_REG_3, 4);
+	/* return 0; */
+	insn_buf[7] = BPF_MOV64_IMM(BPF_REG_0, 0);
+	insn_buf[8] = BPF_JMP_A(5);
+	/* einval: s->cur = s->end = 0; return -EINVAL; */
+	insn_buf[9] = BPF_ST_MEM(BPF_DW, BPF_REG_1, 0, 0);
+	insn_buf[10] = BPF_MOV64_IMM(BPF_REG_0, -EINVAL);
+	insn_buf[11] = BPF_JMP_A(2);
+	/* e2big: s->cur = s->end = 0; return -E2BIG; */
+	insn_buf[12] = BPF_ST_MEM(BPF_DW, BPF_REG_1, 0, 0);
+	insn_buf[13] = BPF_MOV64_IMM(BPF_REG_0, -E2BIG);
+
+	return 14;
+}
+
 int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 		     struct bpf_insn *insn_buf, int insn_idx, int *cnt)
 {
@@ -19850,6 +19886,8 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 		insn_buf[4] = BPF_ALU64_REG(BPF_SUB, BPF_REG_0, BPF_REG_1);
 		insn_buf[5] = BPF_ALU64_IMM(BPF_NEG, BPF_REG_0, 0);
 		*cnt = 6;
+	} else if (desc->func_id == special_kfunc_list[KF_bpf_iter_num_new]) {
+		*cnt = inline_bpf_iter_num_new(insn_buf);
 	}
 
 	if (env->insn_aux_data[insn_idx].arg_prog) {
-- 
2.53.0-Meta

[PATCH bpf-next v2 2/4] bpf: Inline bpf_iter_num_next() kfunc

[Puranjay Mohan]

bpf_iter_num_next() is called on every iteration of a bpf_for() loop and
is the hot path of the numeric open-coded iterator. It only advances the
on-stack iterator state and returns a pointer to it, so open-coding it in
the verifier removes a function call from each loop iteration.

Inline it in bpf_fixup_kfunc_call() by replacing the call with an
equivalent instruction sequence. R1 holds the pointer to the on-stack
bpf_iter_num; the returned pointer to s->cur is R1 itself since s->cur is
the first member.

s->cur and s->end are int, so the kfunc's (s64)(s->cur + 1) >= s->end
test is equivalent to a signed 32-bit comparison of (s->cur + 1) against
s->end: s->cur + 1 is computed as a 32-bit int in the kfunc as well, and
sign-extending both sides of a comparison of two int values does not
change its result. The inlined code therefore uses a 32-bit compare and
needs no sign extension.

Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
---
 kernel/bpf/verifier.c | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)

diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 9b9748b2a7cf5..832785b0670f5 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -19757,6 +19757,33 @@ static int inline_bpf_iter_num_new(struct bpf_insn *insn_buf)
 	return 14;
 }
 
+/*
+ * Inline bpf_iter_num_next(). R1 holds the pointer to the iterator. Keep in
+ * sync with the kfunc in kernel/bpf/bpf_iter.c.
+ */
+static int inline_bpf_iter_num_next(struct bpf_insn *insn_buf)
+{
+	/*
+	 * s->cur and s->end are int, so the (s64)(s->cur + 1) >= s->end check
+	 * is equivalent to a signed 32-bit comparison of (s->cur + 1) against
+	 * s->end and needs no sign extension.
+	 */
+	insn_buf[0] = BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1, 0);
+	insn_buf[1] = BPF_ALU32_IMM(BPF_ADD, BPF_REG_0, 1);
+	insn_buf[2] = BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, 4);
+	/* if ((s32)(s->cur + 1) >= (s32)s->end) goto done; */
+	insn_buf[3] = BPF_JMP32_REG(BPF_JSGE, BPF_REG_0, BPF_REG_2, 3);
+	/* s->cur++; return &s->cur; */
+	insn_buf[4] = BPF_STX_MEM(BPF_W, BPF_REG_1, BPF_REG_0, 0);
+	insn_buf[5] = BPF_MOV64_REG(BPF_REG_0, BPF_REG_1);
+	insn_buf[6] = BPF_JMP_A(2);
+	/* done: s->cur = s->end = 0; return NULL; */
+	insn_buf[7] = BPF_ST_MEM(BPF_DW, BPF_REG_1, 0, 0);
+	insn_buf[8] = BPF_MOV64_IMM(BPF_REG_0, 0);
+
+	return 9;
+}
+
 int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 		     struct bpf_insn *insn_buf, int insn_idx, int *cnt)
 {
@@ -19888,6 +19915,8 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 		*cnt = 6;
 	} else if (desc->func_id == special_kfunc_list[KF_bpf_iter_num_new]) {
 		*cnt = inline_bpf_iter_num_new(insn_buf);
+	} else if (desc->func_id == special_kfunc_list[KF_bpf_iter_num_next]) {
+		*cnt = inline_bpf_iter_num_next(insn_buf);
 	}
 
 	if (env->insn_aux_data[insn_idx].arg_prog) {
-- 
2.53.0-Meta

[PATCH bpf-next v2 3/4] bpf: Inline bpf_iter_num_destroy() kfunc

[Puranjay Mohan]

bpf_iter_num_destroy() only zeroes the on-stack iterator state, which is
a single 8-byte store. Open-code it in the verifier to drop the call
emitted at the end of every bpf_for() loop.

Inline it in bpf_fixup_kfunc_call() by replacing the call with a store of
zero to the iterator, matching the kfunc which clears both s->cur and
s->end. R1 holds the pointer to the on-stack bpf_iter_num.

Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
---
 kernel/bpf/verifier.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 832785b0670f5..64108bc9d461c 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -19784,6 +19784,18 @@ static int inline_bpf_iter_num_next(struct bpf_insn *insn_buf)
 	return 9;
 }
 
+/*
+ * Inline bpf_iter_num_destroy(). R1 holds the pointer to the iterator. Keep in
+ * sync with the kfunc in kernel/bpf/bpf_iter.c.
+ */
+static int inline_bpf_iter_num_destroy(struct bpf_insn *insn_buf)
+{
+	/* s->cur = s->end = 0; */
+	insn_buf[0] = BPF_ST_MEM(BPF_DW, BPF_REG_1, 0, 0);
+
+	return 1;
+}
+
 int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 		     struct bpf_insn *insn_buf, int insn_idx, int *cnt)
 {
@@ -19917,6 +19929,8 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 		*cnt = inline_bpf_iter_num_new(insn_buf);
 	} else if (desc->func_id == special_kfunc_list[KF_bpf_iter_num_next]) {
 		*cnt = inline_bpf_iter_num_next(insn_buf);
+	} else if (desc->func_id == special_kfunc_list[KF_bpf_iter_num_destroy]) {
+		*cnt = inline_bpf_iter_num_destroy(insn_buf);
 	}
 
 	if (env->insn_aux_data[insn_idx].arg_prog) {
-- 
2.53.0-Meta

[PATCH bpf-next v2 4/4] selftests/bpf: Add bpf_for() benchmark

[Puranjay Mohan]

The bpf_for() macro builds on the numeric open-coded iterator kfuncs
bpf_iter_num_{new,next,destroy}(). Add a benchmark, modelled on the
existing bpf_loop benchmark, that repeatedly runs a bpf_for() loop so the
per-iteration cost of the iterator can be measured, e.g. to quantify the
effect of inlining the iterator kfuncs in the verifier.

The BPF program runs an inner bpf_for(i, 0, nr_loops) loop with an empty
body 1000 times per trigger and accounts nr_loops hits per outer
iteration, mirroring bench_bpf_loop so the two are directly comparable.

nr_loops defaults to 1000 so that the per-iteration bpf_iter_num_next()
cost, rather than the one-time bpf_iter_num_new()/destroy() setup and
teardown, dominates the reported numbers:

  $ ./bench -p 1 --nr_loops 1000 bpf-for

Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
---
 tools/testing/selftests/bpf/Makefile          |   2 +
 tools/testing/selftests/bpf/bench.c           |   4 +
 .../selftests/bpf/benchs/bench_bpf_for.c      | 105 ++++++++++++++++++
 .../selftests/bpf/benchs/run_bench_bpf_for.sh |  15 +++
 .../selftests/bpf/progs/bpf_for_bench.c       |  32 ++++++
 5 files changed, 158 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/benchs/bench_bpf_for.c
 create mode 100755 tools/testing/selftests/bpf/benchs/run_bench_bpf_for.sh
 create mode 100644 tools/testing/selftests/bpf/progs/bpf_for_bench.c

diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile
index b642ee489ea64..00b6ff636e3c1 100644
--- a/tools/testing/selftests/bpf/Makefile
+++ b/tools/testing/selftests/bpf/Makefile
@@ -967,6 +967,7 @@ $(OUTPUT)/bench_ringbufs.o: $(OUTPUT)/ringbuf_bench.skel.h \
 			    $(OUTPUT)/perfbuf_bench.skel.h
 $(OUTPUT)/bench_bloom_filter_map.o: $(OUTPUT)/bloom_filter_bench.skel.h
 $(OUTPUT)/bench_bpf_loop.o: $(OUTPUT)/bpf_loop_bench.skel.h
+$(OUTPUT)/bench_bpf_for.o: $(OUTPUT)/bpf_for_bench.skel.h
 $(OUTPUT)/bench_strncmp.o: $(OUTPUT)/strncmp_bench.skel.h
 $(OUTPUT)/bench_bpf_hashmap_full_update.o: $(OUTPUT)/bpf_hashmap_full_update_bench.skel.h
 $(OUTPUT)/bench_local_storage.o: $(OUTPUT)/local_storage_bench.skel.h
@@ -992,6 +993,7 @@ $(OUTPUT)/bench: $(OUTPUT)/bench.o \
 		 $(OUTPUT)/bench_ringbufs.o \
 		 $(OUTPUT)/bench_bloom_filter_map.o \
 		 $(OUTPUT)/bench_bpf_loop.o \
+		 $(OUTPUT)/bench_bpf_for.o \
 		 $(OUTPUT)/bench_strncmp.o \
 		 $(OUTPUT)/bench_bpf_hashmap_full_update.o \
 		 $(OUTPUT)/bench_local_storage.o \
diff --git a/tools/testing/selftests/bpf/bench.c b/tools/testing/selftests/bpf/bench.c
index 3d9d2cd7764bd..b86b73456d3ca 100644
--- a/tools/testing/selftests/bpf/bench.c
+++ b/tools/testing/selftests/bpf/bench.c
@@ -276,6 +276,7 @@ static const struct argp_option opts[] = {
 extern struct argp bench_ringbufs_argp;
 extern struct argp bench_bloom_map_argp;
 extern struct argp bench_bpf_loop_argp;
+extern struct argp bench_bpf_for_argp;
 extern struct argp bench_local_storage_argp;
 extern struct argp bench_local_storage_rcu_tasks_trace_argp;
 extern struct argp bench_strncmp_argp;
@@ -292,6 +293,7 @@ static const struct argp_child bench_parsers[] = {
 	{ &bench_ringbufs_argp, 0, "Ring buffers benchmark", 0 },
 	{ &bench_bloom_map_argp, 0, "Bloom filter map benchmark", 0 },
 	{ &bench_bpf_loop_argp, 0, "bpf_loop helper benchmark", 0 },
+	{ &bench_bpf_for_argp, 0, "bpf_for loop benchmark", 0 },
 	{ &bench_local_storage_argp, 0, "local_storage benchmark", 0 },
 	{ &bench_strncmp_argp, 0, "bpf_strncmp helper benchmark", 0 },
 	{ &bench_local_storage_rcu_tasks_trace_argp, 0,
@@ -557,6 +559,7 @@ extern const struct bench bench_bloom_false_positive;
 extern const struct bench bench_hashmap_without_bloom;
 extern const struct bench bench_hashmap_with_bloom;
 extern const struct bench bench_bpf_loop;
+extern const struct bench bench_bpf_for;
 extern const struct bench bench_strncmp_no_helper;
 extern const struct bench bench_strncmp_helper;
 extern const struct bench bench_bpf_hashmap_full_update;
@@ -640,6 +643,7 @@ static const struct bench *benchs[] = {
 	&bench_hashmap_without_bloom,
 	&bench_hashmap_with_bloom,
 	&bench_bpf_loop,
+	&bench_bpf_for,
 	&bench_strncmp_no_helper,
 	&bench_strncmp_helper,
 	&bench_bpf_hashmap_full_update,
diff --git a/tools/testing/selftests/bpf/benchs/bench_bpf_for.c b/tools/testing/selftests/bpf/benchs/bench_bpf_for.c
new file mode 100644
index 0000000000000..789f8bfe25b9a
--- /dev/null
+++ b/tools/testing/selftests/bpf/benchs/bench_bpf_for.c
@@ -0,0 +1,105 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+#include <argp.h>
+#include "bench.h"
+#include "bpf_for_bench.skel.h"
+
+/* BPF triggering benchmarks */
+static struct ctx {
+	struct bpf_for_bench *skel;
+} ctx;
+
+static struct {
+	__u32 nr_loops;
+} args = {
+	/*
+	 * Default to a large loop count so the per-iteration
+	 * bpf_iter_num_next() cost dominates the one-time
+	 * bpf_iter_num_new()/destroy() setup and teardown.
+	 */
+	.nr_loops = 1000,
+};
+
+enum {
+	ARG_NR_LOOPS = 4000,
+};
+
+static const struct argp_option opts[] = {
+	{ "nr_loops", ARG_NR_LOOPS, "nr_loops", 0,
+		"Set number of iterations for the bpf_for() loop"},
+	{},
+};
+
+static error_t parse_arg(int key, char *arg, struct argp_state *state)
+{
+	switch (key) {
+	case ARG_NR_LOOPS:
+		args.nr_loops = strtol(arg, NULL, 10);
+		break;
+	default:
+		return ARGP_ERR_UNKNOWN;
+	}
+
+	return 0;
+}
+
+/* exported into benchmark runner */
+const struct argp bench_bpf_for_argp = {
+	.options = opts,
+	.parser = parse_arg,
+};
+
+static void validate(void)
+{
+	if (env.consumer_cnt != 0) {
+		fprintf(stderr, "benchmark doesn't support consumer!\n");
+		exit(1);
+	}
+}
+
+static void *producer(void *input)
+{
+	while (true)
+		/* trigger the bpf program */
+		syscall(__NR_getpgid);
+
+	return NULL;
+}
+
+static void measure(struct bench_res *res)
+{
+	res->hits = atomic_swap(&ctx.skel->bss->hits, 0);
+}
+
+static void setup(void)
+{
+	struct bpf_link *link;
+
+	setup_libbpf();
+
+	ctx.skel = bpf_for_bench__open_and_load();
+	if (!ctx.skel) {
+		fprintf(stderr, "failed to open skeleton\n");
+		exit(1);
+	}
+
+	link = bpf_program__attach(ctx.skel->progs.benchmark);
+	if (!link) {
+		fprintf(stderr, "failed to attach program!\n");
+		exit(1);
+	}
+
+	ctx.skel->bss->nr_loops = args.nr_loops;
+}
+
+const struct bench bench_bpf_for = {
+	.name = "bpf-for",
+	.argp = &bench_bpf_for_argp,
+	.validate = validate,
+	.setup = setup,
+	.producer_thread = producer,
+	.measure = measure,
+	.report_progress = ops_report_progress,
+	.report_final = ops_report_final,
+};
diff --git a/tools/testing/selftests/bpf/benchs/run_bench_bpf_for.sh b/tools/testing/selftests/bpf/benchs/run_bench_bpf_for.sh
new file mode 100755
index 0000000000000..7da6453920dab
--- /dev/null
+++ b/tools/testing/selftests/bpf/benchs/run_bench_bpf_for.sh
@@ -0,0 +1,15 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+
+source ./benchs/run_common.sh
+
+set -eufo pipefail
+
+for t in 1 4 8 12 16; do
+for i in 10 100 500 1000 5000 10000 50000 100000 500000 1000000; do
+subtitle "nr_loops: $i, nr_threads: $t"
+	summarize_ops "bpf_for: " \
+	    "$($RUN_BENCH -p $t --nr_loops $i bpf-for)"
+	printf "\n"
+done
+done
diff --git a/tools/testing/selftests/bpf/progs/bpf_for_bench.c b/tools/testing/selftests/bpf/progs/bpf_for_bench.c
new file mode 100644
index 0000000000000..920f8f188e864
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/bpf_for_bench.c
@@ -0,0 +1,32 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+#include "vmlinux.h"
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+char _license[] SEC("license") = "GPL";
+
+u32 nr_loops;
+long hits;
+
+static int outer_loop(__u32 index, void *data)
+{
+	int i;
+
+	/*
+	 * Empty body: the work being measured is the open-coded numeric
+	 * iterator itself (bpf_iter_num_new/next/destroy behind bpf_for()).
+	 */
+	bpf_for(i, 0, nr_loops)
+		;
+	__sync_add_and_fetch(&hits, nr_loops);
+	return 0;
+}
+
+SEC("fentry/" SYS_PREFIX "sys_getpgid")
+int benchmark(void *ctx)
+{
+	bpf_loop(1000, outer_loop, NULL, 0);
+	return 0;
+}
-- 
2.53.0-Meta

Re: [PATCH bpf-next v2 4/4] selftests/bpf: Add bpf_for() benchmark

[Eduard Zingerman]

On Fri, 2026-07-17 at 05:02 -0700, Puranjay Mohan wrote:
> The bpf_for() macro builds on the numeric open-coded iterator kfuncs
> bpf_iter_num_{new,next,destroy}(). Add a benchmark, modelled on the
> existing bpf_loop benchmark, that repeatedly runs a bpf_for() loop so the
> per-iteration cost of the iterator can be measured, e.g. to quantify the
> effect of inlining the iterator kfuncs in the verifier.
> 
> The BPF program runs an inner bpf_for(i, 0, nr_loops) loop with an empty
> body 1000 times per trigger and accounts nr_loops hits per outer
> iteration, mirroring bench_bpf_loop so the two are directly comparable.
> 
> nr_loops defaults to 1000 so that the per-iteration bpf_iter_num_next()
> cost, rather than the one-time bpf_iter_num_new()/destroy() setup and
> teardown, dominates the reported numbers:
> 
>   $ ./bench -p 1 --nr_loops 1000 bpf-for
> 
> Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
> ---

Could you please add a few tests with __xlated tag, e.g. as in:

  tools/testing/selftests/bpf/progs/epilogue_exit.c

To verify the shape of the inlined code?

Re: [PATCH bpf-next v2 1/4] bpf: Inline bpf_iter_num_new() kfunc

[Eduard Zingerman]

On Fri, 2026-07-17 at 05:02 -0700, Puranjay Mohan wrote:

...

> +static int inline_bpf_iter_num_new(struct bpf_insn *insn_buf)
> +{
> +	/* if (start > end) goto einval; */
> +	insn_buf[0] = BPF_JMP32_REG(BPF_JSGT, BPF_REG_2, BPF_REG_3, 8);
> +	/*
> +	 * start <= end here, so the range end - start fits in a u32; compute it
> +	 * as a 32-bit subtraction that zero-extends into r0.
> +	 */
> +	insn_buf[1] = BPF_MOV32_REG(BPF_REG_0, BPF_REG_3);
> +	insn_buf[2] = BPF_ALU32_REG(BPF_SUB, BPF_REG_0, BPF_REG_2);
> +	/* if (r0 > BPF_MAX_LOOPS) goto e2big; */
> +	insn_buf[3] = BPF_JMP_IMM(BPF_JSGT, BPF_REG_0, BPF_MAX_LOOPS, 8);
> +	/* s->cur = start - 1; */
> +	insn_buf[4] = BPF_ALU32_IMM(BPF_ADD, BPF_REG_2, -1);
> +	insn_buf[5] = BPF_STX_MEM(BPF_W, BPF_REG_1, BPF_REG_2, 0);
> +	/* s->end = end; */
> +	insn_buf[6] = BPF_STX_MEM(BPF_W, BPF_REG_1, BPF_REG_3, 4);
> +	/* return 0; */
> +	insn_buf[7] = BPF_MOV64_IMM(BPF_REG_0, 0);
> +	insn_buf[8] = BPF_JMP_A(5);
> +	/* einval: s->cur = s->end = 0; return -EINVAL; */
> +	insn_buf[9] = BPF_ST_MEM(BPF_DW, BPF_REG_1, 0, 0);
> +	insn_buf[10] = BPF_MOV64_IMM(BPF_REG_0, -EINVAL);
> +	insn_buf[11] = BPF_JMP_A(2);
> +	/* e2big: s->cur = s->end = 0; return -E2BIG; */
> +	insn_buf[12] = BPF_ST_MEM(BPF_DW, BPF_REG_1, 0, 0);
> +	insn_buf[13] = BPF_MOV64_IMM(BPF_REG_0, -E2BIG);
> +
> +	return 14;
> +}

The translation lgtm, do we want to avoid two checks at the start in
case when r2 and r3 are known statically?

Re: [PATCH bpf-next v2 2/4] bpf: Inline bpf_iter_num_next() kfunc

[Eduard Zingerman]

On Fri, 2026-07-17 at 05:02 -0700, Puranjay Mohan wrote:

...

> +static int inline_bpf_iter_num_next(struct bpf_insn *insn_buf)
> +{
> +	/*
> +	 * s->cur and s->end are int, so the (s64)(s->cur + 1) >= s->end check
> +	 * is equivalent to a signed 32-bit comparison of (s->cur + 1) against
> +	 * s->end and needs no sign extension.
> +	 */
> +	insn_buf[0] = BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1, 0);
> +	insn_buf[1] = BPF_ALU32_IMM(BPF_ADD, BPF_REG_0, 1);
> +	insn_buf[2] = BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, 4);
> +	/* if ((s32)(s->cur + 1) >= (s32)s->end) goto done; */
> +	insn_buf[3] = BPF_JMP32_REG(BPF_JSGE, BPF_REG_0, BPF_REG_2, 3);
> +	/* s->cur++; return &s->cur; */
> +	insn_buf[4] = BPF_STX_MEM(BPF_W, BPF_REG_1, BPF_REG_0, 0);
> +	insn_buf[5] = BPF_MOV64_REG(BPF_REG_0, BPF_REG_1);
> +	insn_buf[6] = BPF_JMP_A(2);
> +	/* done: s->cur = s->end = 0; return NULL; */
> +	insn_buf[7] = BPF_ST_MEM(BPF_DW, BPF_REG_1, 0, 0);
> +	insn_buf[8] = BPF_MOV64_IMM(BPF_REG_0, 0);
> +
> +	return 9;
> +}

Nit: let's have this (and others) as:

  int i = 0;

  insn_buf[i++] = ...
  insn_buf[i++] = ...
  insn_buf[i++] = ...
  return i;

The translation lgtm.

...

Re: [PATCH bpf-next v2 3/4] bpf: Inline bpf_iter_num_destroy() kfunc

[Eduard Zingerman]

On Fri, 2026-07-17 at 05:02 -0700, Puranjay Mohan wrote:
> bpf_iter_num_destroy() only zeroes the on-stack iterator state, which is
> a single 8-byte store. Open-code it in the verifier to drop the call
> emitted at the end of every bpf_for() loop.
> 
> Inline it in bpf_fixup_kfunc_call() by replacing the call with a store of
> zero to the iterator, matching the kfunc which clears both s->cur and
> s->end. R1 holds the pointer to the on-stack bpf_iter_num.
> 
> Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
> ---

Acked-by: Eduard Zingerman <eddyz87@gmail.com>

Re: [PATCH bpf-next v2 4/4] selftests/bpf: Add bpf_for() benchmark

[Eduard Zingerman]

On Fri, 2026-07-17 at 05:02 -0700, Puranjay Mohan wrote:

...

> diff --git a/tools/testing/selftests/bpf/progs/bpf_for_bench.c b/tools/testing/selftests/bpf/progs/bpf_for_bench.c
> new file mode 100644
> index 0000000000000..920f8f188e864
> --- /dev/null
> +++ b/tools/testing/selftests/bpf/progs/bpf_for_bench.c
> @@ -0,0 +1,32 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
> +
> +#include "vmlinux.h"
> +#include <bpf/bpf_helpers.h>
> +#include "bpf_misc.h"
> +
> +char _license[] SEC("license") = "GPL";
> +
> +u32 nr_loops;
> +long hits;
> +
> +static int outer_loop(__u32 index, void *data)
> +{
> +	int i;
> +
> +	/*
> +	 * Empty body: the work being measured is the open-coded numeric
> +	 * iterator itself (bpf_iter_num_new/next/destroy behind bpf_for()).
> +	 */
> +	bpf_for(i, 0, nr_loops)
> +		;

I wonder how does this compare with regular for loop that also reads
it's counter.

> +	__sync_add_and_fetch(&hits, nr_loops);
> +	return 0;
> +}
> +
> +SEC("fentry/" SYS_PREFIX "sys_getpgid")
> +int benchmark(void *ctx)
> +{
> +	bpf_loop(1000, outer_loop, NULL, 0);
> +	return 0;
> +}