[PATCH bpf-next v2 0/4] Switch from kfuncs to direct helper calls in prologue/epilogue

[PATCH bpf-next v2 0/4] Switch from kfuncs to direct helper calls in prologue/epilogue

[Jakub Sitnicki]

This series enables direct helper calls using BPF_EMIT_CALL from prologue
and epilogue code generated by verifier ops. The goal is to simplify the
calling convention and remove kfunc support from prologue/epilogue, as
suggested by Alexei [1].

Patch 1 adds the infrastructure to mark direct helper calls as finalized
(already resolved) so the verifier skips the imm fixup.

Patch 2 converts bpf_qdisc to use BPF_EMIT_CALL instead of BPF_CALL_KFUNC
for the init prologue and reset/destroy epilogue helpers.

Patch 3 removes the corresponding selftests that exercised kfuncs in
prologue/epilogue.

Patch 4 removes the now-unused kfunc support code from prologue/epilogue
handling in the verifier.

[1] https://lore.kernel.org/bpf/CAADnVQJ=kmVAZsgkG9P2nEBTUG3E4PrDG=Yz8tfeFysH4ZBqVw@mail.gmail.com/

Signed-off-by: Jakub Sitnicki <jakub@cloudflare.com>
---
Changes in v2:
- Swap patch 3 & 4 to avoid selftests fails on bisect (Eduard)
- Rename is_epilogue flag to finalize_helper_calls (Eduard)
- Untie mark_helper_calls_finalized call from epilogue_idx (Eduard)
- Drop review tags for patch 1 after tweaks in patch_insn_buf (patch 1)
- s/__bpf_base_call/__bpf_call_base/ in doc comment (patch 1)
- Link to v1: https://lore.kernel.org/r/20260119-skb-meta-bpf-emit-call-from-prologue-v1-0-e8b88d6430d8@cloudflare.com

---
Jakub Sitnicki (4):
      bpf, verifier: Support direct helper calls from prologue/epilogue
      bpf: net_sched: Use direct helper calls instead of kfuncs in pro/epilogue
      selftests/bpf: Remove tests for prologue/epilogue with kfuncs
      bpf: Remove kfunc support in prologue and epilogue

 include/linux/bpf_verifier.h                       |  1 +
 kernel/bpf/verifier.c                              | 52 ++++++------
 net/core/filter.c                                  |  3 +-
 net/sched/bpf_qdisc.c                              | 76 ++++++++----------
 .../selftests/bpf/prog_tests/pro_epilogue.c        |  2 -
 .../selftests/bpf/progs/pro_epilogue_with_kfunc.c  | 88 ---------------------
 .../testing/selftests/bpf/test_kmods/bpf_testmod.c | 92 ----------------------
 7 files changed, 63 insertions(+), 251 deletions(-)

[PATCH bpf-next v2 1/4] bpf, verifier: Support direct helper calls from prologue/epilogue

[Jakub Sitnicki]

Prepare to remove support for calling kfuncs from prologue & epilogue.

Instead allow direct helpers calls using BPF_EMIT_CALL. Such calls already
contain helper offset relative to __bpf_call_base and must bypass the
verifier's patch_call_imm fixup, which expects BPF helper IDs rather than a
pre-resolved offsets.

Add a finalized_call flag to bpf_insn_aux_data to mark call instructions
with resolved offsets so the verifier can skip patch_call_imm fixup for
these calls.

Note that the target of BPF_EMIT_CALL should be wrapped with BPF_CALL_x to
prevent an ABI mismatch between BPF and C on 32-bit architectures.

Suggested-by: Alexei Starovoitov <ast@kernel.org>
Signed-off-by: Jakub Sitnicki <jakub@cloudflare.com>
---
 include/linux/bpf_verifier.h |  1 +
 kernel/bpf/verifier.c        | 29 ++++++++++++++++++++++++++---
 net/core/filter.c            |  3 +--
 3 files changed, 28 insertions(+), 5 deletions(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 8355b585cd18..1ccbb47aca2d 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -561,6 +561,7 @@ struct bpf_insn_aux_data {
 	bool non_sleepable; /* helper/kfunc may be called from non-sleepable context */
 	bool is_iter_next; /* bpf_iter_<type>_next() kfunc call */
 	bool call_with_percpu_alloc_ptr; /* {this,per}_cpu_ptr() with prog percpu alloc */
+	bool finalized_call; /* call holds resolved helper offset relative to __bpf_call_base */
 	u8 alu_state; /* used in combination with alu_limit */
 	/* true if STX or LDX instruction is a part of a spill/fill
 	 * pattern for a bpf_fastcall call.
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index c7f5234d5fd2..0b4feedc14c9 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -22201,6 +22201,19 @@ static int opt_subreg_zext_lo32_rnd_hi32(struct bpf_verifier_env *env,
 	return 0;
 }
 
+/* Mark helper calls within prog->insns[off ... off+cnt-1] range as resolved,
+ * meaning imm contains the helper offset. Used for prologue & epilogue.
+ */
+static void mark_helper_calls_finalized(struct bpf_verifier_env *env, int off, int cnt)
+{
+	int i;
+
+	for (i = 0; i < cnt; i++) {
+		if (bpf_helper_call(&env->prog->insnsi[i + off]))
+			env->insn_aux_data[i + off].finalized_call = true;
+	}
+}
+
 /* convert load instructions that access fields of a context type into a
  * sequence of instructions that access fields of the underlying structure:
  *     struct __sk_buff    -> struct sk_buff
@@ -22267,6 +22280,8 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env)
 			ret = add_kfunc_in_insns(env, insn_buf, cnt - 1);
 			if (ret < 0)
 				return ret;
+
+			mark_helper_calls_finalized(env, 0, cnt - 1);
 		}
 	}
 
@@ -22280,6 +22295,7 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env)
 
 	for (i = 0; i < insn_cnt; i++, insn++) {
 		bpf_convert_ctx_access_t convert_ctx_access;
+		bool finalize_helper_calls = false;
 		u8 mode;
 
 		if (env->insn_aux_data[i + delta].nospec) {
@@ -22346,6 +22362,7 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env)
 				 * epilogue.
 				 */
 				epilogue_idx = i + delta;
+				finalize_helper_calls = true;
 			}
 			goto patch_insn_buf;
 		} else {
@@ -22495,12 +22512,14 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env)
 		new_prog = bpf_patch_insn_data(env, i + delta, insn_buf, cnt);
 		if (!new_prog)
 			return -ENOMEM;
+		env->prog = new_prog;
 
-		delta += cnt - 1;
+		if (finalize_helper_calls)
+			mark_helper_calls_finalized(env, i + delta, cnt - 1);
 
 		/* keep walking new program and skip insns we just inserted */
-		env->prog = new_prog;
-		insn      = new_prog->insnsi + i + delta;
+		delta += cnt - 1;
+		insn = new_prog->insnsi + i + delta;
 	}
 
 	return 0;
@@ -23909,6 +23928,9 @@ static int do_misc_fixups(struct bpf_verifier_env *env)
 			goto next_insn;
 		}
 patch_call_imm:
+		if (env->insn_aux_data[i + delta].finalized_call)
+			goto next_insn;
+
 		fn = env->ops->get_func_proto(insn->imm, env->prog);
 		/* all functions that have prototype and verifier allowed
 		 * programs to call them, must be real in-kernel functions
@@ -23920,6 +23942,7 @@ static int do_misc_fixups(struct bpf_verifier_env *env)
 			return -EFAULT;
 		}
 		insn->imm = fn->func - __bpf_call_base;
+		env->insn_aux_data[i + delta].finalized_call = true;
 next_insn:
 		if (subprogs[cur_subprog + 1].start == i + delta + 1) {
 			subprogs[cur_subprog].stack_depth += stack_depth_extra;
diff --git a/net/core/filter.c b/net/core/filter.c
index d14401193b01..cb39388f69a9 100644
--- a/net/core/filter.c
+++ b/net/core/filter.c
@@ -9082,8 +9082,7 @@ static int bpf_unclone_prologue(struct bpf_insn *insn_buf, bool direct_write,
 	/* ret = bpf_skb_pull_data(skb, 0); */
 	*insn++ = BPF_MOV64_REG(BPF_REG_6, BPF_REG_1);
 	*insn++ = BPF_ALU64_REG(BPF_XOR, BPF_REG_2, BPF_REG_2);
-	*insn++ = BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
-			       BPF_FUNC_skb_pull_data);
+	*insn++ = BPF_EMIT_CALL(bpf_skb_pull_data);
 	/* if (!ret)
 	 *      goto restore;
 	 * return TC_ACT_SHOT;

-- 
2.43.0

[PATCH bpf-next v2 2/4] bpf: net_sched: Use direct helper calls instead of kfuncs in pro/epilogue

[Jakub Sitnicki]

Convert bpf_qdisc prologue and epilogue to use BPF_EMIT_CALL for direct
helper calls instead of BPF_CALL_KFUNC.

Remove the BTF_ID_LIST entries for these functions since they are no longer
registered as kfuncs.

Reviewed-by: Eduard Zingerman <eddyz87@gmail.com>
Acked-by: Martin KaFai Lau <martin.lau@kernel.org>
Reviewed-by: Amery Hung <ameryhung@gmail.com>
Signed-off-by: Jakub Sitnicki <jakub@cloudflare.com>
---
 net/sched/bpf_qdisc.c | 76 ++++++++++++++++++++++++---------------------------
 1 file changed, 35 insertions(+), 41 deletions(-)

diff --git a/net/sched/bpf_qdisc.c b/net/sched/bpf_qdisc.c
index 098ca02aed89..cad9701d3b95 100644
--- a/net/sched/bpf_qdisc.c
+++ b/net/sched/bpf_qdisc.c
@@ -130,7 +130,30 @@ static int bpf_qdisc_btf_struct_access(struct bpf_verifier_log *log,
 	return 0;
 }
 
-BTF_ID_LIST_SINGLE(bpf_qdisc_init_prologue_ids, func, bpf_qdisc_init_prologue)
+/* bpf_qdisc_init_prologue - Called in prologue of .init. */
+BPF_CALL_2(bpf_qdisc_init_prologue, struct Qdisc *, sch,
+	   struct netlink_ext_ack *, extack)
+{
+	struct bpf_sched_data *q = qdisc_priv(sch);
+	struct net_device *dev = qdisc_dev(sch);
+	struct Qdisc *p;
+
+	qdisc_watchdog_init(&q->watchdog, sch);
+
+	if (sch->parent != TC_H_ROOT) {
+		/* If qdisc_lookup() returns NULL, it means .init is called by
+		 * qdisc_create_dflt() in mq/mqprio_init and the parent qdisc
+		 * has not been added to qdisc_hash yet.
+		 */
+		p = qdisc_lookup(dev, TC_H_MAJ(sch->parent));
+		if (p && !(p->flags & TCQ_F_MQROOT)) {
+			NL_SET_ERR_MSG(extack, "BPF qdisc only supported on root or mq");
+			return -EINVAL;
+		}
+	}
+
+	return 0;
+}
 
 static int bpf_qdisc_gen_prologue(struct bpf_insn *insn_buf, bool direct_write,
 				  const struct bpf_prog *prog)
@@ -151,7 +174,7 @@ static int bpf_qdisc_gen_prologue(struct bpf_insn *insn_buf, bool direct_write,
 	*insn++ = BPF_MOV64_REG(BPF_REG_6, BPF_REG_1);
 	*insn++ = BPF_LDX_MEM(BPF_DW, BPF_REG_2, BPF_REG_1, 16);
 	*insn++ = BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, 0);
-	*insn++ = BPF_CALL_KFUNC(0, bpf_qdisc_init_prologue_ids[0]);
+	*insn++ = BPF_EMIT_CALL(bpf_qdisc_init_prologue);
 	*insn++ = BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 1);
 	*insn++ = BPF_EXIT_INSN();
 	*insn++ = BPF_MOV64_REG(BPF_REG_1, BPF_REG_6);
@@ -160,7 +183,15 @@ static int bpf_qdisc_gen_prologue(struct bpf_insn *insn_buf, bool direct_write,
 	return insn - insn_buf;
 }
 
-BTF_ID_LIST_SINGLE(bpf_qdisc_reset_destroy_epilogue_ids, func, bpf_qdisc_reset_destroy_epilogue)
+/* bpf_qdisc_reset_destroy_epilogue - Called in epilogue of .reset and .destroy */
+BPF_CALL_1(bpf_qdisc_reset_destroy_epilogue, struct Qdisc *, sch)
+{
+	struct bpf_sched_data *q = qdisc_priv(sch);
+
+	qdisc_watchdog_cancel(&q->watchdog);
+
+	return 0;
+}
 
 static int bpf_qdisc_gen_epilogue(struct bpf_insn *insn_buf, const struct bpf_prog *prog,
 				  s16 ctx_stack_off)
@@ -178,7 +209,7 @@ static int bpf_qdisc_gen_epilogue(struct bpf_insn *insn_buf, const struct bpf_pr
 	 */
 	*insn++ = BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_FP, ctx_stack_off);
 	*insn++ = BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, 0);
-	*insn++ = BPF_CALL_KFUNC(0, bpf_qdisc_reset_destroy_epilogue_ids[0]);
+	*insn++ = BPF_EMIT_CALL(bpf_qdisc_reset_destroy_epilogue);
 	*insn++ = BPF_EXIT_INSN();
 
 	return insn - insn_buf;
@@ -230,41 +261,6 @@ __bpf_kfunc void bpf_qdisc_watchdog_schedule(struct Qdisc *sch, u64 expire, u64
 	qdisc_watchdog_schedule_range_ns(&q->watchdog, expire, delta_ns);
 }
 
-/* bpf_qdisc_init_prologue - Hidden kfunc called in prologue of .init. */
-__bpf_kfunc int bpf_qdisc_init_prologue(struct Qdisc *sch,
-					struct netlink_ext_ack *extack)
-{
-	struct bpf_sched_data *q = qdisc_priv(sch);
-	struct net_device *dev = qdisc_dev(sch);
-	struct Qdisc *p;
-
-	qdisc_watchdog_init(&q->watchdog, sch);
-
-	if (sch->parent != TC_H_ROOT) {
-		/* If qdisc_lookup() returns NULL, it means .init is called by
-		 * qdisc_create_dflt() in mq/mqprio_init and the parent qdisc
-		 * has not been added to qdisc_hash yet.
-		 */
-		p = qdisc_lookup(dev, TC_H_MAJ(sch->parent));
-		if (p && !(p->flags & TCQ_F_MQROOT)) {
-			NL_SET_ERR_MSG(extack, "BPF qdisc only supported on root or mq");
-			return -EINVAL;
-		}
-	}
-
-	return 0;
-}
-
-/* bpf_qdisc_reset_destroy_epilogue - Hidden kfunc called in epilogue of .reset
- * and .destroy
- */
-__bpf_kfunc void bpf_qdisc_reset_destroy_epilogue(struct Qdisc *sch)
-{
-	struct bpf_sched_data *q = qdisc_priv(sch);
-
-	qdisc_watchdog_cancel(&q->watchdog);
-}
-
 /* bpf_qdisc_bstats_update - Update Qdisc basic statistics
  * @sch: The qdisc from which an skb is dequeued.
  * @skb: The skb to be dequeued.
@@ -282,8 +278,6 @@ BTF_ID_FLAGS(func, bpf_kfree_skb, KF_RELEASE)
 BTF_ID_FLAGS(func, bpf_qdisc_skb_drop, KF_RELEASE)
 BTF_ID_FLAGS(func, bpf_dynptr_from_skb)
 BTF_ID_FLAGS(func, bpf_qdisc_watchdog_schedule)
-BTF_ID_FLAGS(func, bpf_qdisc_init_prologue)
-BTF_ID_FLAGS(func, bpf_qdisc_reset_destroy_epilogue)
 BTF_ID_FLAGS(func, bpf_qdisc_bstats_update)
 BTF_KFUNCS_END(qdisc_kfunc_ids)
 

-- 
2.43.0

[PATCH bpf-next v2 3/4] selftests/bpf: Remove tests for prologue/epilogue with kfuncs

[Jakub Sitnicki]

Remove pro_epilogue_with_kfunc test program and its supporting code in
bpf_testmod. This test exercised calling kfuncs from prologue and epilogue,
which is no longer supported after the switch to direct helper calls.

Reviewed-by: Eduard Zingerman <eddyz87@gmail.com>
Acked-by: Martin KaFai Lau <martin.lau@kernel.org>
Reviewed-by: Amery Hung <ameryhung@gmail.com>
Signed-off-by: Jakub Sitnicki <jakub@cloudflare.com>
---
 .../selftests/bpf/prog_tests/pro_epilogue.c        |  2 -
 .../selftests/bpf/progs/pro_epilogue_with_kfunc.c  | 88 ---------------------
 .../testing/selftests/bpf/test_kmods/bpf_testmod.c | 92 ----------------------
 3 files changed, 182 deletions(-)

diff --git a/tools/testing/selftests/bpf/prog_tests/pro_epilogue.c b/tools/testing/selftests/bpf/prog_tests/pro_epilogue.c
index 5d3c00a08a88..509883e6823a 100644
--- a/tools/testing/selftests/bpf/prog_tests/pro_epilogue.c
+++ b/tools/testing/selftests/bpf/prog_tests/pro_epilogue.c
@@ -6,7 +6,6 @@
 #include "epilogue_tailcall.skel.h"
 #include "pro_epilogue_goto_start.skel.h"
 #include "epilogue_exit.skel.h"
-#include "pro_epilogue_with_kfunc.skel.h"
 
 struct st_ops_args {
 	__u64 a;
@@ -56,7 +55,6 @@ void test_pro_epilogue(void)
 	RUN_TESTS(pro_epilogue);
 	RUN_TESTS(pro_epilogue_goto_start);
 	RUN_TESTS(epilogue_exit);
-	RUN_TESTS(pro_epilogue_with_kfunc);
 	if (test__start_subtest("tailcall"))
 		test_tailcall();
 }
diff --git a/tools/testing/selftests/bpf/progs/pro_epilogue_with_kfunc.c b/tools/testing/selftests/bpf/progs/pro_epilogue_with_kfunc.c
deleted file mode 100644
index a5a8f08ac8fb..000000000000
--- a/tools/testing/selftests/bpf/progs/pro_epilogue_with_kfunc.c
+++ /dev/null
@@ -1,88 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0
-/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */
-
-#include <vmlinux.h>
-#include <bpf/bpf_tracing.h>
-#include "bpf_misc.h"
-#include "../test_kmods/bpf_testmod.h"
-#include "../test_kmods/bpf_testmod_kfunc.h"
-
-char _license[] SEC("license") = "GPL";
-
-void __kfunc_btf_root(void)
-{
-	bpf_kfunc_st_ops_inc10(NULL);
-}
-
-static __noinline __used int subprog(struct st_ops_args *args)
-{
-	args->a += 1;
-	return args->a;
-}
-
-__success
-/* prologue */
-__xlated("0: r8 = r1")
-__xlated("1: r1 = 0")
-__xlated("2: call kernel-function")
-__xlated("3: if r0 != 0x0 goto pc+5")
-__xlated("4: r6 = *(u64 *)(r8 +0)")
-__xlated("5: r7 = *(u64 *)(r6 +0)")
-__xlated("6: r7 += 1000")
-__xlated("7: *(u64 *)(r6 +0) = r7")
-__xlated("8: goto pc+2")
-__xlated("9: r1 = r0")
-__xlated("10: call kernel-function")
-__xlated("11: r1 = r8")
-/* save __u64 *ctx to stack */
-__xlated("12: *(u64 *)(r10 -8) = r1")
-/* main prog */
-__xlated("13: r1 = *(u64 *)(r1 +0)")
-__xlated("14: r6 = r1")
-__xlated("15: call kernel-function")
-__xlated("16: r1 = r6")
-__xlated("17: call pc+")
-/* epilogue */
-__xlated("18: r1 = 0")
-__xlated("19: r6 = 0")
-__xlated("20: call kernel-function")
-__xlated("21: if r0 != 0x0 goto pc+6")
-__xlated("22: r1 = *(u64 *)(r10 -8)")
-__xlated("23: r1 = *(u64 *)(r1 +0)")
-__xlated("24: r6 = *(u64 *)(r1 +0)")
-__xlated("25: r6 += 10000")
-__xlated("26: *(u64 *)(r1 +0) = r6")
-__xlated("27: goto pc+2")
-__xlated("28: r1 = r0")
-__xlated("29: call kernel-function")
-__xlated("30: r0 = r6")
-__xlated("31: r0 *= 2")
-__xlated("32: exit")
-SEC("struct_ops/test_pro_epilogue")
-__naked int test_kfunc_pro_epilogue(void)
-{
-	asm volatile (
-	"r1 = *(u64 *)(r1 +0);"
-	"r6 = r1;"
-	"call %[bpf_kfunc_st_ops_inc10];"
-	"r1 = r6;"
-	"call subprog;"
-	"exit;"
-	:
-	: __imm(bpf_kfunc_st_ops_inc10)
-	: __clobber_all);
-}
-
-SEC("syscall")
-__retval(22022) /* (PROLOGUE_A [1000] + KFUNC_INC10 + SUBPROG_A [1] + EPILOGUE_A [10000]) * 2 */
-int syscall_pro_epilogue(void *ctx)
-{
-	struct st_ops_args args = {};
-
-	return bpf_kfunc_st_ops_test_pro_epilogue(&args);
-}
-
-SEC(".struct_ops.link")
-struct bpf_testmod_st_ops pro_epilogue_with_kfunc = {
-	.test_pro_epilogue = (void *)test_kfunc_pro_epilogue,
-};
diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
index 77a81fa8ec6a..5d611f186f23 100644
--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
@@ -1429,85 +1429,6 @@ static int bpf_test_mod_st_ops__test_pro_epilogue(struct st_ops_args *args)
 	return 0;
 }
 
-static int bpf_cgroup_from_id_id;
-static int bpf_cgroup_release_id;
-
-static int st_ops_gen_prologue_with_kfunc(struct bpf_insn *insn_buf, bool direct_write,
-					  const struct bpf_prog *prog)
-{
-	struct bpf_insn *insn = insn_buf;
-
-	/* r8 = r1; // r8 will be "u64 *ctx".
-	 * r1 = 0;
-	 * r0 = bpf_cgroup_from_id(r1);
-	 * if r0 != 0 goto pc+5;
-	 * r6 = r8[0]; // r6 will be "struct st_ops *args".
-	 * r7 = r6->a;
-	 * r7 += 1000;
-	 * r6->a = r7;
-	 * goto pc+2;
-	 * r1 = r0;
-	 * bpf_cgroup_release(r1);
-	 * r1 = r8;
-	 */
-	*insn++ = BPF_MOV64_REG(BPF_REG_8, BPF_REG_1);
-	*insn++ = BPF_MOV64_IMM(BPF_REG_1, 0);
-	*insn++ = BPF_CALL_KFUNC(0, bpf_cgroup_from_id_id);
-	*insn++ = BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 5);
-	*insn++ = BPF_LDX_MEM(BPF_DW, BPF_REG_6, BPF_REG_8, 0);
-	*insn++ = BPF_LDX_MEM(BPF_DW, BPF_REG_7, BPF_REG_6, offsetof(struct st_ops_args, a));
-	*insn++ = BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, 1000);
-	*insn++ = BPF_STX_MEM(BPF_DW, BPF_REG_6, BPF_REG_7, offsetof(struct st_ops_args, a));
-	*insn++ = BPF_JMP_IMM(BPF_JA, 0, 0, 2);
-	*insn++ = BPF_MOV64_REG(BPF_REG_1, BPF_REG_0);
-	*insn++ = BPF_CALL_KFUNC(0, bpf_cgroup_release_id);
-	*insn++ = BPF_MOV64_REG(BPF_REG_1, BPF_REG_8);
-	*insn++ = prog->insnsi[0];
-
-	return insn - insn_buf;
-}
-
-static int st_ops_gen_epilogue_with_kfunc(struct bpf_insn *insn_buf, const struct bpf_prog *prog,
-					  s16 ctx_stack_off)
-{
-	struct bpf_insn *insn = insn_buf;
-
-	/* r1 = 0;
-	 * r6 = 0;
-	 * r0 = bpf_cgroup_from_id(r1);
-	 * if r0 != 0 goto pc+6;
-	 * r1 = stack[ctx_stack_off]; // r1 will be "u64 *ctx"
-	 * r1 = r1[0]; // r1 will be "struct st_ops *args"
-	 * r6 = r1->a;
-	 * r6 += 10000;
-	 * r1->a = r6;
-	 * goto pc+2
-	 * r1 = r0;
-	 * bpf_cgroup_release(r1);
-	 * r0 = r6;
-	 * r0 *= 2;
-	 * BPF_EXIT;
-	 */
-	*insn++ = BPF_MOV64_IMM(BPF_REG_1, 0);
-	*insn++ = BPF_MOV64_IMM(BPF_REG_6, 0);
-	*insn++ = BPF_CALL_KFUNC(0, bpf_cgroup_from_id_id);
-	*insn++ = BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 6);
-	*insn++ = BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_FP, ctx_stack_off);
-	*insn++ = BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, 0);
-	*insn++ = BPF_LDX_MEM(BPF_DW, BPF_REG_6, BPF_REG_1, offsetof(struct st_ops_args, a));
-	*insn++ = BPF_ALU64_IMM(BPF_ADD, BPF_REG_6, 10000);
-	*insn++ = BPF_STX_MEM(BPF_DW, BPF_REG_1, BPF_REG_6, offsetof(struct st_ops_args, a));
-	*insn++ = BPF_JMP_IMM(BPF_JA, 0, 0, 2);
-	*insn++ = BPF_MOV64_REG(BPF_REG_1, BPF_REG_0);
-	*insn++ = BPF_CALL_KFUNC(0, bpf_cgroup_release_id);
-	*insn++ = BPF_MOV64_REG(BPF_REG_0, BPF_REG_6);
-	*insn++ = BPF_ALU64_IMM(BPF_MUL, BPF_REG_0, 2);
-	*insn++ = BPF_EXIT_INSN();
-
-	return insn - insn_buf;
-}
-
-#define KFUNC_PRO_EPI_PREFIX "test_kfunc_"
 static int st_ops_gen_prologue(struct bpf_insn *insn_buf, bool direct_write,
 			       const struct bpf_prog *prog)
 {
@@ -1517,9 +1438,6 @@ static int st_ops_gen_prologue(struct bpf_insn *insn_buf, bool direct_write,
 	    strcmp(prog->aux->attach_func_name, "test_pro_epilogue"))
 		return 0;
 
-	if (!strncmp(prog->aux->name, KFUNC_PRO_EPI_PREFIX, strlen(KFUNC_PRO_EPI_PREFIX)))
-		return st_ops_gen_prologue_with_kfunc(insn_buf, direct_write, prog);
-
 	/* r6 = r1[0]; // r6 will be "struct st_ops *args". r1 is "u64 *ctx".
 	 * r7 = r6->a;
 	 * r7 += 1000;
@@ -1543,9 +1461,6 @@ static int st_ops_gen_epilogue(struct bpf_insn *insn_buf, const struct bpf_prog
 	    strcmp(prog->aux->attach_func_name, "test_pro_epilogue"))
 		return 0;
 
-	if (!strncmp(prog->aux->name, KFUNC_PRO_EPI_PREFIX, strlen(KFUNC_PRO_EPI_PREFIX)))
-		return st_ops_gen_epilogue_with_kfunc(insn_buf, prog, ctx_stack_off);
-
 	/* r1 = stack[ctx_stack_off]; // r1 will be "u64 *ctx"
 	 * r1 = r1[0]; // r1 will be "struct st_ops *args"
 	 * r6 = r1->a;
@@ -1616,13 +1531,6 @@ static void st_ops_unreg(void *kdata, struct bpf_link *link)
 
 static int st_ops_init(struct btf *btf)
 {
-	struct btf *kfunc_btf;
-
-	bpf_cgroup_from_id_id = bpf_find_btf_id("bpf_cgroup_from_id", BTF_KIND_FUNC, &kfunc_btf);
-	bpf_cgroup_release_id = bpf_find_btf_id("bpf_cgroup_release", BTF_KIND_FUNC, &kfunc_btf);
-	if (bpf_cgroup_from_id_id < 0 || bpf_cgroup_release_id < 0)
-		return -EINVAL;
-
 	return 0;
 }
 

-- 
2.43.0

[PATCH bpf-next v2 4/4] bpf: Remove kfunc support in prologue and epilogue

[Jakub Sitnicki]

Remove add_kfunc_in_insns() and its call sites in convert_ctx_accesses().
This function was used to register kfuncs found in prologue and epilogue
instructions, but is no longer needed now that we use direct helper calls
via BPF_EMIT_CALL instead.

Reviewed-by: Eduard Zingerman <eddyz87@gmail.com>
Reviewed-by: Amery Hung <ameryhung@gmail.com>
Signed-off-by: Jakub Sitnicki <jakub@cloudflare.com>
---
 kernel/bpf/verifier.c | 25 +------------------------
 1 file changed, 1 insertion(+), 24 deletions(-)

diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 0b4feedc14c9..41ff0c459a66 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -3543,21 +3543,6 @@ bpf_jit_find_kfunc_model(const struct bpf_prog *prog,
 	return res ? &res->func_model : NULL;
 }
 
-static int add_kfunc_in_insns(struct bpf_verifier_env *env,
-			      struct bpf_insn *insn, int cnt)
-{
-	int i, ret;
-
-	for (i = 0; i < cnt; i++, insn++) {
-		if (bpf_pseudo_kfunc_call(insn)) {
-			ret = add_kfunc_call(env, insn->imm, insn->off);
-			if (ret < 0)
-				return ret;
-		}
-	}
-	return 0;
-}
-
 static int add_subprog_and_kfunc(struct bpf_verifier_env *env)
 {
 	struct bpf_subprog_info *subprog = env->subprog_info;
@@ -22223,7 +22208,7 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env)
 {
 	struct bpf_subprog_info *subprogs = env->subprog_info;
 	const struct bpf_verifier_ops *ops = env->ops;
-	int i, cnt, size, ctx_field_size, ret, delta = 0, epilogue_cnt = 0;
+	int i, cnt, size, ctx_field_size, delta = 0, epilogue_cnt = 0;
 	const int insn_cnt = env->prog->len;
 	struct bpf_insn *epilogue_buf = env->epilogue_buf;
 	struct bpf_insn *insn_buf = env->insn_buf;
@@ -22252,10 +22237,6 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env)
 				return -ENOMEM;
 			env->prog = new_prog;
 			delta += cnt - 1;
-
-			ret = add_kfunc_in_insns(env, epilogue_buf, epilogue_cnt - 1);
-			if (ret < 0)
-				return ret;
 		}
 	}
 
@@ -22277,10 +22258,6 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env)
 			env->prog = new_prog;
 			delta += cnt - 1;
 
-			ret = add_kfunc_in_insns(env, insn_buf, cnt - 1);
-			if (ret < 0)
-				return ret;
-
 			mark_helper_calls_finalized(env, 0, cnt - 1);
 		}
 	}

-- 
2.43.0

Re: [PATCH bpf-next v2 0/4] Switch from kfuncs to direct helper calls in prologue/epilogue

[patchwork-bot+netdevbpf]

Hello:

This series was applied to bpf/bpf-next.git (master)
by Alexei Starovoitov <ast@kernel.org>:

On Fri, 23 Jan 2026 18:05:17 +0100 you wrote:
> This series enables direct helper calls using BPF_EMIT_CALL from prologue
> and epilogue code generated by verifier ops. The goal is to simplify the
> calling convention and remove kfunc support from prologue/epilogue, as
> suggested by Alexei [1].
> 
> Patch 1 adds the infrastructure to mark direct helper calls as finalized
> (already resolved) so the verifier skips the imm fixup.
> 
> [...]

Here is the summary with links:
  - [bpf-next,v2,1/4] bpf, verifier: Support direct helper calls from prologue/epilogue
    https://git.kernel.org/bpf/bpf-next/c/af46ae41da7c
  - [bpf-next,v2,2/4] bpf: net_sched: Use direct helper calls instead of kfuncs in pro/epilogue
    https://git.kernel.org/bpf/bpf-next/c/5c2106d44168
  - [bpf-next,v2,3/4] selftests/bpf: Remove tests for prologue/epilogue with kfuncs
    https://git.kernel.org/bpf/bpf-next/c/d3ddc749df28
  - [bpf-next,v2,4/4] bpf: Remove kfunc support in prologue and epilogue
    https://git.kernel.org/bpf/bpf-next/c/68d23a660c4f

You are awesome, thank you!
-- 
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html

Re: [PATCH bpf-next v2 1/4] bpf, verifier: Support direct helper calls from prologue/epilogue

[Martin KaFai Lau]

On 1/23/26 9:05 AM, Jakub Sitnicki wrote:
> @@ -23909,6 +23928,9 @@ static int do_misc_fixups(struct bpf_verifier_env *env)
>   			goto next_insn;
>   		}
>   patch_call_imm:
> +		if (env->insn_aux_data[i + delta].finalized_call)
> +			goto next_insn;
> +
>   		fn = env->ops->get_func_proto(insn->imm, env->prog);
>   		/* all functions that have prototype and verifier allowed
>   		 * programs to call them, must be real in-kernel functions
> @@ -23920,6 +23942,7 @@ static int do_misc_fixups(struct bpf_verifier_env *env)
>   			return -EFAULT;
>   		}
>   		insn->imm = fn->func - __bpf_call_base;
> +		env->insn_aux_data[i + delta].finalized_call = true;
>   next_insn:
>   		if (subprogs[cur_subprog + 1].start == i + delta + 1) {
>   			subprogs[cur_subprog].stack_depth += stack_depth_extra;
> diff --git a/net/core/filter.c b/net/core/filter.c
> index d14401193b01..cb39388f69a9 100644
> --- a/net/core/filter.c
> +++ b/net/core/filter.c
> @@ -9082,8 +9082,7 @@ static int bpf_unclone_prologue(struct bpf_insn *insn_buf, bool direct_write,
>   	/* ret = bpf_skb_pull_data(skb, 0); */
>   	*insn++ = BPF_MOV64_REG(BPF_REG_6, BPF_REG_1);
>   	*insn++ = BPF_ALU64_REG(BPF_XOR, BPF_REG_2, BPF_REG_2);
> -	*insn++ = BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
> -			       BPF_FUNC_skb_pull_data);
> +	*insn++ = BPF_EMIT_CALL(bpf_skb_pull_data);

Ihor reported that the test_map has started failing on arm64.

https://github.com/kernel-patches/bpf/actions/runs/21298510237/job/61311053284
https://github.com/kernel-patches/bpf/actions/runs/21298505282/job/61312363930
https://github.com/kernel-patches/bpf/actions/runs/21301695907/job/61321375157

For BPF_PROG_TYPE_SK_SKB, the BPF_FUNC_skb_pull_data has a different
func_proto. It is sk_skb_pull_data instead of bpf_skb_pull_data.
A different func needs to be emitted here based on prog type.
Not sure why it only fails on arm64.

The set has been reverted to get the CI going.
Please address the issue, add a test for this case, and
then respin. Thanks.

Re: [PATCH bpf-next v2 1/4] bpf, verifier: Support direct helper calls from prologue/epilogue

[Jakub Sitnicki]

On Fri, Jan 23, 2026 at 03:49 PM -08, Martin KaFai Lau wrote:
> On 1/23/26 9:05 AM, Jakub Sitnicki wrote:
>> @@ -23909,6 +23928,9 @@ static int do_misc_fixups(struct bpf_verifier_env *env)
>>   			goto next_insn;
>>   		}
>>   patch_call_imm:
>> +		if (env->insn_aux_data[i + delta].finalized_call)
>> +			goto next_insn;
>> +
>>   		fn = env->ops->get_func_proto(insn->imm, env->prog);
>>   		/* all functions that have prototype and verifier allowed
>>   		 * programs to call them, must be real in-kernel functions
>> @@ -23920,6 +23942,7 @@ static int do_misc_fixups(struct bpf_verifier_env *env)
>>   			return -EFAULT;
>>   		}
>>   		insn->imm = fn->func - __bpf_call_base;
>> +		env->insn_aux_data[i + delta].finalized_call = true;
>>   next_insn:
>>   		if (subprogs[cur_subprog + 1].start == i + delta + 1) {
>>   			subprogs[cur_subprog].stack_depth += stack_depth_extra;
>> diff --git a/net/core/filter.c b/net/core/filter.c
>> index d14401193b01..cb39388f69a9 100644
>> --- a/net/core/filter.c
>> +++ b/net/core/filter.c
>> @@ -9082,8 +9082,7 @@ static int bpf_unclone_prologue(struct bpf_insn *insn_buf, bool direct_write,
>>   	/* ret = bpf_skb_pull_data(skb, 0); */
>>   	*insn++ = BPF_MOV64_REG(BPF_REG_6, BPF_REG_1);
>>   	*insn++ = BPF_ALU64_REG(BPF_XOR, BPF_REG_2, BPF_REG_2);
>> -	*insn++ = BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
>> -			       BPF_FUNC_skb_pull_data);
>> +	*insn++ = BPF_EMIT_CALL(bpf_skb_pull_data);
>
> Ihor reported that the test_map has started failing on arm64.
>
> https://github.com/kernel-patches/bpf/actions/runs/21298510237/job/61311053284
> https://github.com/kernel-patches/bpf/actions/runs/21298505282/job/61312363930
> https://github.com/kernel-patches/bpf/actions/runs/21301695907/job/61321375157
>
> For BPF_PROG_TYPE_SK_SKB, the BPF_FUNC_skb_pull_data has a different
> func_proto. It is sk_skb_pull_data instead of bpf_skb_pull_data.
> A different func needs to be emitted here based on prog type.
> Not sure why it only fails on arm64.
>
> The set has been reverted to get the CI going.
> Please address the issue, add a test for this case, and
> then respin. Thanks.

Sorry for this oversight.

I see now that sk_skb_pull_data doesn't bpf_compute_data_pointers which
writes over skb->cb. Perhaps that has something to do with the failure.

Will get this sorted.

Expect some delay. This is a pre-conferece week (FOSDEM).