[PATCH bpf-next 0/4] Switch from kfuncs to direct helper calls in prologue/epilogue

[PATCH bpf-next 0/4] Switch from kfuncs to direct helper calls in prologue/epilogue

[Jakub Sitnicki]

This series enables direct helper calls using BPF_EMIT_CALL from prologue
and epilogue code generated by verifier ops. The goal is to simplify the
calling convention and remove kfunc support from prologue/epilogue, as
suggested by Alexei [1].

Patch 1 adds the infrastructure to mark direct helper calls as finalized
(already resolved) so the verifier skips the imm fixup.

Patch 2 converts bpf_qdisc to use BPF_EMIT_CALL instead of BPF_CALL_KFUNC
for the init prologue and reset/destroy epilogue helpers.

Patch 3 removes the now-unused kfunc support code from prologue/epilogue
handling in the verifier.

Patch 4 removes the corresponding selftests that exercised kfuncs in
prologue/epilogue.

[1] https://lore.kernel.org/bpf/CAADnVQJ=kmVAZsgkG9P2nEBTUG3E4PrDG=Yz8tfeFysH4ZBqVw@mail.gmail.com/

Signed-off-by: Jakub Sitnicki <jakub@cloudflare.com>
---
Jakub Sitnicki (4):
      bpf, verifier: Support direct helper calls from prologue/epilogue
      bpf: net_sched: Use direct helper calls instead of kfuncs in pro/epilogue
      bpf: Remove kfunc support in prologue and epilogue
      selftests/bpf: Remove tests for prologue/epilogue with kfuncs

 include/linux/bpf_verifier.h                       |  1 +
 kernel/bpf/verifier.c                              | 47 +++++------
 net/core/filter.c                                  |  3 +-
 net/sched/bpf_qdisc.c                              | 76 ++++++++----------
 .../selftests/bpf/prog_tests/pro_epilogue.c        |  2 -
 .../selftests/bpf/progs/pro_epilogue_with_kfunc.c  | 88 ---------------------
 .../testing/selftests/bpf/test_kmods/bpf_testmod.c | 92 ----------------------
 7 files changed, 61 insertions(+), 248 deletions(-)

[PATCH bpf-next 1/4] bpf, verifier: Support direct helper calls from prologue/epilogue

[Jakub Sitnicki]

Prepare to remove support for calling kfuncs from prologue & epilogue.

Instead allow direct helpers calls using BPF_EMIT_CALL. Such calls already
contain helper offset relative to __bpf_call_base and must bypass the
verifier's patch_call_imm fixup, which expects BPF helper IDs rather than a
pre-resolved offsets.

Add a finalized_call flag to bpf_insn_aux_data to mark call instructions
with resolved offsets so the verifier can skip patch_call_imm fixup for
these calls.

Note that the target of BPF_EMIT_CALL should be wrapped with BPF_CALL_x to
prevent an ABI mismatch between BPF and C on 32-bit architectures.

Suggested-by: Alexei Starovoitov <ast@kernel.org>
Signed-off-by: Jakub Sitnicki <jakub@cloudflare.com>
---
 include/linux/bpf_verifier.h |  1 +
 kernel/bpf/verifier.c        | 24 ++++++++++++++++++++++++
 net/core/filter.c            |  3 +--
 3 files changed, 26 insertions(+), 2 deletions(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 130bcbd66f60..e358f01c300d 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -561,6 +561,7 @@ struct bpf_insn_aux_data {
 	bool non_sleepable; /* helper/kfunc may be called from non-sleepable context */
 	bool is_iter_next; /* bpf_iter_<type>_next() kfunc call */
 	bool call_with_percpu_alloc_ptr; /* {this,per}_cpu_ptr() with prog percpu alloc */
+	bool finalized_call; /* call holds resolved helper offset relative to __bpf_base_call */
 	u8 alu_state; /* used in combination with alu_limit */
 	/* true if STX or LDX instruction is a part of a spill/fill
 	 * pattern for a bpf_fastcall call.
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 9de0ec0c3ed9..15694a40ca02 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -21801,6 +21801,19 @@ static int opt_subreg_zext_lo32_rnd_hi32(struct bpf_verifier_env *env,
 	return 0;
 }
 
+/* Mark helper calls within prog->insns[off ... off+cnt-1] range as resolved,
+ * meaning imm contains the helper offset. Used for prologue & epilogue.
+ */
+static void mark_helper_calls_finalized(struct bpf_verifier_env *env, int off, int cnt)
+{
+	int i;
+
+	for (i = 0; i < cnt; i++) {
+		if (bpf_helper_call(&env->prog->insnsi[i + off]))
+			env->insn_aux_data[i + off].finalized_call = true;
+	}
+}
+
 /* convert load instructions that access fields of a context type into a
  * sequence of instructions that access fields of the underlying structure:
  *     struct __sk_buff    -> struct sk_buff
@@ -21867,6 +21880,8 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env)
 			ret = add_kfunc_in_insns(env, insn_buf, cnt - 1);
 			if (ret < 0)
 				return ret;
+
+			mark_helper_calls_finalized(env, 0, cnt - 1);
 		}
 	}
 
@@ -21880,6 +21895,7 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env)
 
 	for (i = 0; i < insn_cnt; i++, insn++) {
 		bpf_convert_ctx_access_t convert_ctx_access;
+		bool is_epilogue = false;
 		u8 mode;
 
 		if (env->insn_aux_data[i + delta].nospec) {
@@ -21946,6 +21962,7 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env)
 				 * epilogue.
 				 */
 				epilogue_idx = i + delta;
+				is_epilogue = true;
 			}
 			goto patch_insn_buf;
 		} else {
@@ -22101,6 +22118,9 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env)
 		/* keep walking new program and skip insns we just inserted */
 		env->prog = new_prog;
 		insn      = new_prog->insnsi + i + delta;
+
+		if (is_epilogue)
+			mark_helper_calls_finalized(env, epilogue_idx, cnt - 1);
 	}
 
 	return 0;
@@ -23477,6 +23497,9 @@ static int do_misc_fixups(struct bpf_verifier_env *env)
 			goto next_insn;
 		}
 patch_call_imm:
+		if (env->insn_aux_data[i + delta].finalized_call)
+			goto next_insn;
+
 		fn = env->ops->get_func_proto(insn->imm, env->prog);
 		/* all functions that have prototype and verifier allowed
 		 * programs to call them, must be real in-kernel functions
@@ -23488,6 +23511,7 @@ static int do_misc_fixups(struct bpf_verifier_env *env)
 			return -EFAULT;
 		}
 		insn->imm = fn->func - __bpf_call_base;
+		env->insn_aux_data[i + delta].finalized_call = true;
 next_insn:
 		if (subprogs[cur_subprog + 1].start == i + delta + 1) {
 			subprogs[cur_subprog].stack_depth += stack_depth_extra;
diff --git a/net/core/filter.c b/net/core/filter.c
index d43df98e1ded..aa0fabcd21d1 100644
--- a/net/core/filter.c
+++ b/net/core/filter.c
@@ -9082,8 +9082,7 @@ static int bpf_unclone_prologue(struct bpf_insn *insn_buf, bool direct_write,
 	/* ret = bpf_skb_pull_data(skb, 0); */
 	*insn++ = BPF_MOV64_REG(BPF_REG_6, BPF_REG_1);
 	*insn++ = BPF_ALU64_REG(BPF_XOR, BPF_REG_2, BPF_REG_2);
-	*insn++ = BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
-			       BPF_FUNC_skb_pull_data);
+	*insn++ = BPF_EMIT_CALL(bpf_skb_pull_data);
 	/* if (!ret)
 	 *      goto restore;
 	 * return TC_ACT_SHOT;

-- 
2.43.0

[PATCH bpf-next 2/4] bpf: net_sched: Use direct helper calls instead of kfuncs in pro/epilogue

[Jakub Sitnicki]

Convert bpf_qdisc prologue and epilogue to use BPF_EMIT_CALL for direct
helper calls instead of BPF_CALL_KFUNC.

Remove the BTF_ID_LIST entries for these functions since they are no longer
registered as kfuncs.

Signed-off-by: Jakub Sitnicki <jakub@cloudflare.com>
---
 net/sched/bpf_qdisc.c | 76 ++++++++++++++++++++++++---------------------------
 1 file changed, 35 insertions(+), 41 deletions(-)

diff --git a/net/sched/bpf_qdisc.c b/net/sched/bpf_qdisc.c
index 098ca02aed89..cad9701d3b95 100644
--- a/net/sched/bpf_qdisc.c
+++ b/net/sched/bpf_qdisc.c
@@ -130,7 +130,30 @@ static int bpf_qdisc_btf_struct_access(struct bpf_verifier_log *log,
 	return 0;
 }
 
-BTF_ID_LIST_SINGLE(bpf_qdisc_init_prologue_ids, func, bpf_qdisc_init_prologue)
+/* bpf_qdisc_init_prologue - Called in prologue of .init. */
+BPF_CALL_2(bpf_qdisc_init_prologue, struct Qdisc *, sch,
+	   struct netlink_ext_ack *, extack)
+{
+	struct bpf_sched_data *q = qdisc_priv(sch);
+	struct net_device *dev = qdisc_dev(sch);
+	struct Qdisc *p;
+
+	qdisc_watchdog_init(&q->watchdog, sch);
+
+	if (sch->parent != TC_H_ROOT) {
+		/* If qdisc_lookup() returns NULL, it means .init is called by
+		 * qdisc_create_dflt() in mq/mqprio_init and the parent qdisc
+		 * has not been added to qdisc_hash yet.
+		 */
+		p = qdisc_lookup(dev, TC_H_MAJ(sch->parent));
+		if (p && !(p->flags & TCQ_F_MQROOT)) {
+			NL_SET_ERR_MSG(extack, "BPF qdisc only supported on root or mq");
+			return -EINVAL;
+		}
+	}
+
+	return 0;
+}
 
 static int bpf_qdisc_gen_prologue(struct bpf_insn *insn_buf, bool direct_write,
 				  const struct bpf_prog *prog)
@@ -151,7 +174,7 @@ static int bpf_qdisc_gen_prologue(struct bpf_insn *insn_buf, bool direct_write,
 	*insn++ = BPF_MOV64_REG(BPF_REG_6, BPF_REG_1);
 	*insn++ = BPF_LDX_MEM(BPF_DW, BPF_REG_2, BPF_REG_1, 16);
 	*insn++ = BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, 0);
-	*insn++ = BPF_CALL_KFUNC(0, bpf_qdisc_init_prologue_ids[0]);
+	*insn++ = BPF_EMIT_CALL(bpf_qdisc_init_prologue);
 	*insn++ = BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 1);
 	*insn++ = BPF_EXIT_INSN();
 	*insn++ = BPF_MOV64_REG(BPF_REG_1, BPF_REG_6);
@@ -160,7 +183,15 @@ static int bpf_qdisc_gen_prologue(struct bpf_insn *insn_buf, bool direct_write,
 	return insn - insn_buf;
 }
 
-BTF_ID_LIST_SINGLE(bpf_qdisc_reset_destroy_epilogue_ids, func, bpf_qdisc_reset_destroy_epilogue)
+/* bpf_qdisc_reset_destroy_epilogue - Called in epilogue of .reset and .destroy */
+BPF_CALL_1(bpf_qdisc_reset_destroy_epilogue, struct Qdisc *, sch)
+{
+	struct bpf_sched_data *q = qdisc_priv(sch);
+
+	qdisc_watchdog_cancel(&q->watchdog);
+
+	return 0;
+}
 
 static int bpf_qdisc_gen_epilogue(struct bpf_insn *insn_buf, const struct bpf_prog *prog,
 				  s16 ctx_stack_off)
@@ -178,7 +209,7 @@ static int bpf_qdisc_gen_epilogue(struct bpf_insn *insn_buf, const struct bpf_pr
 	 */
 	*insn++ = BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_FP, ctx_stack_off);
 	*insn++ = BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, 0);
-	*insn++ = BPF_CALL_KFUNC(0, bpf_qdisc_reset_destroy_epilogue_ids[0]);
+	*insn++ = BPF_EMIT_CALL(bpf_qdisc_reset_destroy_epilogue);
 	*insn++ = BPF_EXIT_INSN();
 
 	return insn - insn_buf;
@@ -230,41 +261,6 @@ __bpf_kfunc void bpf_qdisc_watchdog_schedule(struct Qdisc *sch, u64 expire, u64
 	qdisc_watchdog_schedule_range_ns(&q->watchdog, expire, delta_ns);
 }
 
-/* bpf_qdisc_init_prologue - Hidden kfunc called in prologue of .init. */
-__bpf_kfunc int bpf_qdisc_init_prologue(struct Qdisc *sch,
-					struct netlink_ext_ack *extack)
-{
-	struct bpf_sched_data *q = qdisc_priv(sch);
-	struct net_device *dev = qdisc_dev(sch);
-	struct Qdisc *p;
-
-	qdisc_watchdog_init(&q->watchdog, sch);
-
-	if (sch->parent != TC_H_ROOT) {
-		/* If qdisc_lookup() returns NULL, it means .init is called by
-		 * qdisc_create_dflt() in mq/mqprio_init and the parent qdisc
-		 * has not been added to qdisc_hash yet.
-		 */
-		p = qdisc_lookup(dev, TC_H_MAJ(sch->parent));
-		if (p && !(p->flags & TCQ_F_MQROOT)) {
-			NL_SET_ERR_MSG(extack, "BPF qdisc only supported on root or mq");
-			return -EINVAL;
-		}
-	}
-
-	return 0;
-}
-
-/* bpf_qdisc_reset_destroy_epilogue - Hidden kfunc called in epilogue of .reset
- * and .destroy
- */
-__bpf_kfunc void bpf_qdisc_reset_destroy_epilogue(struct Qdisc *sch)
-{
-	struct bpf_sched_data *q = qdisc_priv(sch);
-
-	qdisc_watchdog_cancel(&q->watchdog);
-}
-
 /* bpf_qdisc_bstats_update - Update Qdisc basic statistics
  * @sch: The qdisc from which an skb is dequeued.
  * @skb: The skb to be dequeued.
@@ -282,8 +278,6 @@ BTF_ID_FLAGS(func, bpf_kfree_skb, KF_RELEASE)
 BTF_ID_FLAGS(func, bpf_qdisc_skb_drop, KF_RELEASE)
 BTF_ID_FLAGS(func, bpf_dynptr_from_skb)
 BTF_ID_FLAGS(func, bpf_qdisc_watchdog_schedule)
-BTF_ID_FLAGS(func, bpf_qdisc_init_prologue)
-BTF_ID_FLAGS(func, bpf_qdisc_reset_destroy_epilogue)
 BTF_ID_FLAGS(func, bpf_qdisc_bstats_update)
 BTF_KFUNCS_END(qdisc_kfunc_ids)
 

-- 
2.43.0

[PATCH bpf-next 3/4] bpf: Remove kfunc support in prologue and epilogue

[Jakub Sitnicki]

Remove add_kfunc_in_insns() and its call sites in convert_ctx_accesses().
This function was used to register kfuncs found in prologue and epilogue
instructions, but is no longer needed now that we use direct helper calls
via BPF_EMIT_CALL instead.

Signed-off-by: Jakub Sitnicki <jakub@cloudflare.com>
---
 kernel/bpf/verifier.c | 25 +------------------------
 1 file changed, 1 insertion(+), 24 deletions(-)

diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 15694a40ca02..b04b63c9b777 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -3453,21 +3453,6 @@ bpf_jit_find_kfunc_model(const struct bpf_prog *prog,
 	return res ? &res->func_model : NULL;
 }
 
-static int add_kfunc_in_insns(struct bpf_verifier_env *env,
-			      struct bpf_insn *insn, int cnt)
-{
-	int i, ret;
-
-	for (i = 0; i < cnt; i++, insn++) {
-		if (bpf_pseudo_kfunc_call(insn)) {
-			ret = add_kfunc_call(env, insn->imm, insn->off);
-			if (ret < 0)
-				return ret;
-		}
-	}
-	return 0;
-}
-
 static int add_subprog_and_kfunc(struct bpf_verifier_env *env)
 {
 	struct bpf_subprog_info *subprog = env->subprog_info;
@@ -21823,7 +21808,7 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env)
 {
 	struct bpf_subprog_info *subprogs = env->subprog_info;
 	const struct bpf_verifier_ops *ops = env->ops;
-	int i, cnt, size, ctx_field_size, ret, delta = 0, epilogue_cnt = 0;
+	int i, cnt, size, ctx_field_size, delta = 0, epilogue_cnt = 0;
 	const int insn_cnt = env->prog->len;
 	struct bpf_insn *epilogue_buf = env->epilogue_buf;
 	struct bpf_insn *insn_buf = env->insn_buf;
@@ -21852,10 +21837,6 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env)
 				return -ENOMEM;
 			env->prog = new_prog;
 			delta += cnt - 1;
-
-			ret = add_kfunc_in_insns(env, epilogue_buf, epilogue_cnt - 1);
-			if (ret < 0)
-				return ret;
 		}
 	}
 
@@ -21877,10 +21858,6 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env)
 			env->prog = new_prog;
 			delta += cnt - 1;
 
-			ret = add_kfunc_in_insns(env, insn_buf, cnt - 1);
-			if (ret < 0)
-				return ret;
-
 			mark_helper_calls_finalized(env, 0, cnt - 1);
 		}
 	}

-- 
2.43.0

[PATCH bpf-next 4/4] selftests/bpf: Remove tests for prologue/epilogue with kfuncs

[Jakub Sitnicki]

Remove pro_epilogue_with_kfunc test program and its supporting code in
bpf_testmod. This test exercised calling kfuncs from prologue and epilogue,
which is no longer supported after the switch to direct helper calls.

Signed-off-by: Jakub Sitnicki <jakub@cloudflare.com>
---
 .../selftests/bpf/prog_tests/pro_epilogue.c        |  2 -
 .../selftests/bpf/progs/pro_epilogue_with_kfunc.c  | 88 ---------------------
 .../testing/selftests/bpf/test_kmods/bpf_testmod.c | 92 ----------------------
 3 files changed, 182 deletions(-)

diff --git a/tools/testing/selftests/bpf/prog_tests/pro_epilogue.c b/tools/testing/selftests/bpf/prog_tests/pro_epilogue.c
index 5d3c00a08a88..509883e6823a 100644
--- a/tools/testing/selftests/bpf/prog_tests/pro_epilogue.c
+++ b/tools/testing/selftests/bpf/prog_tests/pro_epilogue.c
@@ -6,7 +6,6 @@
 #include "epilogue_tailcall.skel.h"
 #include "pro_epilogue_goto_start.skel.h"
 #include "epilogue_exit.skel.h"
-#include "pro_epilogue_with_kfunc.skel.h"
 
 struct st_ops_args {
 	__u64 a;
@@ -56,7 +55,6 @@ void test_pro_epilogue(void)
 	RUN_TESTS(pro_epilogue);
 	RUN_TESTS(pro_epilogue_goto_start);
 	RUN_TESTS(epilogue_exit);
-	RUN_TESTS(pro_epilogue_with_kfunc);
 	if (test__start_subtest("tailcall"))
 		test_tailcall();
 }
diff --git a/tools/testing/selftests/bpf/progs/pro_epilogue_with_kfunc.c b/tools/testing/selftests/bpf/progs/pro_epilogue_with_kfunc.c
deleted file mode 100644
index a5a8f08ac8fb..000000000000
--- a/tools/testing/selftests/bpf/progs/pro_epilogue_with_kfunc.c
+++ /dev/null
@@ -1,88 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0
-/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */
-
-#include <vmlinux.h>
-#include <bpf/bpf_tracing.h>
-#include "bpf_misc.h"
-#include "../test_kmods/bpf_testmod.h"
-#include "../test_kmods/bpf_testmod_kfunc.h"
-
-char _license[] SEC("license") = "GPL";
-
-void __kfunc_btf_root(void)
-{
-	bpf_kfunc_st_ops_inc10(NULL);
-}
-
-static __noinline __used int subprog(struct st_ops_args *args)
-{
-	args->a += 1;
-	return args->a;
-}
-
-__success
-/* prologue */
-__xlated("0: r8 = r1")
-__xlated("1: r1 = 0")
-__xlated("2: call kernel-function")
-__xlated("3: if r0 != 0x0 goto pc+5")
-__xlated("4: r6 = *(u64 *)(r8 +0)")
-__xlated("5: r7 = *(u64 *)(r6 +0)")
-__xlated("6: r7 += 1000")
-__xlated("7: *(u64 *)(r6 +0) = r7")
-__xlated("8: goto pc+2")
-__xlated("9: r1 = r0")
-__xlated("10: call kernel-function")
-__xlated("11: r1 = r8")
-/* save __u64 *ctx to stack */
-__xlated("12: *(u64 *)(r10 -8) = r1")
-/* main prog */
-__xlated("13: r1 = *(u64 *)(r1 +0)")
-__xlated("14: r6 = r1")
-__xlated("15: call kernel-function")
-__xlated("16: r1 = r6")
-__xlated("17: call pc+")
-/* epilogue */
-__xlated("18: r1 = 0")
-__xlated("19: r6 = 0")
-__xlated("20: call kernel-function")
-__xlated("21: if r0 != 0x0 goto pc+6")
-__xlated("22: r1 = *(u64 *)(r10 -8)")
-__xlated("23: r1 = *(u64 *)(r1 +0)")
-__xlated("24: r6 = *(u64 *)(r1 +0)")
-__xlated("25: r6 += 10000")
-__xlated("26: *(u64 *)(r1 +0) = r6")
-__xlated("27: goto pc+2")
-__xlated("28: r1 = r0")
-__xlated("29: call kernel-function")
-__xlated("30: r0 = r6")
-__xlated("31: r0 *= 2")
-__xlated("32: exit")
-SEC("struct_ops/test_pro_epilogue")
-__naked int test_kfunc_pro_epilogue(void)
-{
-	asm volatile (
-	"r1 = *(u64 *)(r1 +0);"
-	"r6 = r1;"
-	"call %[bpf_kfunc_st_ops_inc10];"
-	"r1 = r6;"
-	"call subprog;"
-	"exit;"
-	:
-	: __imm(bpf_kfunc_st_ops_inc10)
-	: __clobber_all);
-}
-
-SEC("syscall")
-__retval(22022) /* (PROLOGUE_A [1000] + KFUNC_INC10 + SUBPROG_A [1] + EPILOGUE_A [10000]) * 2 */
-int syscall_pro_epilogue(void *ctx)
-{
-	struct st_ops_args args = {};
-
-	return bpf_kfunc_st_ops_test_pro_epilogue(&args);
-}
-
-SEC(".struct_ops.link")
-struct bpf_testmod_st_ops pro_epilogue_with_kfunc = {
-	.test_pro_epilogue = (void *)test_kfunc_pro_epilogue,
-};
diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
index bc07ce9d5477..1a5c163455de 100644
--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
@@ -1400,85 +1400,6 @@ static int bpf_test_mod_st_ops__test_pro_epilogue(struct st_ops_args *args)
 	return 0;
 }
 
-static int bpf_cgroup_from_id_id;
-static int bpf_cgroup_release_id;
-
-static int st_ops_gen_prologue_with_kfunc(struct bpf_insn *insn_buf, bool direct_write,
-					  const struct bpf_prog *prog)
-{
-	struct bpf_insn *insn = insn_buf;
-
-	/* r8 = r1; // r8 will be "u64 *ctx".
-	 * r1 = 0;
-	 * r0 = bpf_cgroup_from_id(r1);
-	 * if r0 != 0 goto pc+5;
-	 * r6 = r8[0]; // r6 will be "struct st_ops *args".
-	 * r7 = r6->a;
-	 * r7 += 1000;
-	 * r6->a = r7;
-	 * goto pc+2;
-	 * r1 = r0;
-	 * bpf_cgroup_release(r1);
-	 * r1 = r8;
-	 */
-	*insn++ = BPF_MOV64_REG(BPF_REG_8, BPF_REG_1);
-	*insn++ = BPF_MOV64_IMM(BPF_REG_1, 0);
-	*insn++ = BPF_CALL_KFUNC(0, bpf_cgroup_from_id_id);
-	*insn++ = BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 5);
-	*insn++ = BPF_LDX_MEM(BPF_DW, BPF_REG_6, BPF_REG_8, 0);
-	*insn++ = BPF_LDX_MEM(BPF_DW, BPF_REG_7, BPF_REG_6, offsetof(struct st_ops_args, a));
-	*insn++ = BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, 1000);
-	*insn++ = BPF_STX_MEM(BPF_DW, BPF_REG_6, BPF_REG_7, offsetof(struct st_ops_args, a));
-	*insn++ = BPF_JMP_IMM(BPF_JA, 0, 0, 2);
-	*insn++ = BPF_MOV64_REG(BPF_REG_1, BPF_REG_0);
-	*insn++ = BPF_CALL_KFUNC(0, bpf_cgroup_release_id);
-	*insn++ = BPF_MOV64_REG(BPF_REG_1, BPF_REG_8);
-	*insn++ = prog->insnsi[0];
-
-	return insn - insn_buf;
-}
-
-static int st_ops_gen_epilogue_with_kfunc(struct bpf_insn *insn_buf, const struct bpf_prog *prog,
-					  s16 ctx_stack_off)
-{
-	struct bpf_insn *insn = insn_buf;
-
-	/* r1 = 0;
-	 * r6 = 0;
-	 * r0 = bpf_cgroup_from_id(r1);
-	 * if r0 != 0 goto pc+6;
-	 * r1 = stack[ctx_stack_off]; // r1 will be "u64 *ctx"
-	 * r1 = r1[0]; // r1 will be "struct st_ops *args"
-	 * r6 = r1->a;
-	 * r6 += 10000;
-	 * r1->a = r6;
-	 * goto pc+2
-	 * r1 = r0;
-	 * bpf_cgroup_release(r1);
-	 * r0 = r6;
-	 * r0 *= 2;
-	 * BPF_EXIT;
-	 */
-	*insn++ = BPF_MOV64_IMM(BPF_REG_1, 0);
-	*insn++ = BPF_MOV64_IMM(BPF_REG_6, 0);
-	*insn++ = BPF_CALL_KFUNC(0, bpf_cgroup_from_id_id);
-	*insn++ = BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 6);
-	*insn++ = BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_FP, ctx_stack_off);
-	*insn++ = BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, 0);
-	*insn++ = BPF_LDX_MEM(BPF_DW, BPF_REG_6, BPF_REG_1, offsetof(struct st_ops_args, a));
-	*insn++ = BPF_ALU64_IMM(BPF_ADD, BPF_REG_6, 10000);
-	*insn++ = BPF_STX_MEM(BPF_DW, BPF_REG_1, BPF_REG_6, offsetof(struct st_ops_args, a));
-	*insn++ = BPF_JMP_IMM(BPF_JA, 0, 0, 2);
-	*insn++ = BPF_MOV64_REG(BPF_REG_1, BPF_REG_0);
-	*insn++ = BPF_CALL_KFUNC(0, bpf_cgroup_release_id);
-	*insn++ = BPF_MOV64_REG(BPF_REG_0, BPF_REG_6);
-	*insn++ = BPF_ALU64_IMM(BPF_MUL, BPF_REG_0, 2);
-	*insn++ = BPF_EXIT_INSN();
-
-	return insn - insn_buf;
-}
-
-#define KFUNC_PRO_EPI_PREFIX "test_kfunc_"
 static int st_ops_gen_prologue(struct bpf_insn *insn_buf, bool direct_write,
 			       const struct bpf_prog *prog)
 {
@@ -1488,9 +1409,6 @@ static int st_ops_gen_prologue(struct bpf_insn *insn_buf, bool direct_write,
 	    strcmp(prog->aux->attach_func_name, "test_pro_epilogue"))
 		return 0;
 
-	if (!strncmp(prog->aux->name, KFUNC_PRO_EPI_PREFIX, strlen(KFUNC_PRO_EPI_PREFIX)))
-		return st_ops_gen_prologue_with_kfunc(insn_buf, direct_write, prog);
-
 	/* r6 = r1[0]; // r6 will be "struct st_ops *args". r1 is "u64 *ctx".
 	 * r7 = r6->a;
 	 * r7 += 1000;
@@ -1514,9 +1432,6 @@ static int st_ops_gen_epilogue(struct bpf_insn *insn_buf, const struct bpf_prog
 	    strcmp(prog->aux->attach_func_name, "test_pro_epilogue"))
 		return 0;
 
-	if (!strncmp(prog->aux->name, KFUNC_PRO_EPI_PREFIX, strlen(KFUNC_PRO_EPI_PREFIX)))
-		return st_ops_gen_epilogue_with_kfunc(insn_buf, prog, ctx_stack_off);
-
 	/* r1 = stack[ctx_stack_off]; // r1 will be "u64 *ctx"
 	 * r1 = r1[0]; // r1 will be "struct st_ops *args"
 	 * r6 = r1->a;
@@ -1587,13 +1502,6 @@ static void st_ops_unreg(void *kdata, struct bpf_link *link)
 
 static int st_ops_init(struct btf *btf)
 {
-	struct btf *kfunc_btf;
-
-	bpf_cgroup_from_id_id = bpf_find_btf_id("bpf_cgroup_from_id", BTF_KIND_FUNC, &kfunc_btf);
-	bpf_cgroup_release_id = bpf_find_btf_id("bpf_cgroup_release", BTF_KIND_FUNC, &kfunc_btf);
-	if (bpf_cgroup_from_id_id < 0 || bpf_cgroup_release_id < 0)
-		return -EINVAL;
-
 	return 0;
 }
 

-- 
2.43.0

Re: [PATCH bpf-next 1/4] bpf, verifier: Support direct helper calls from prologue/epilogue

[Eduard Zingerman]

On Mon, 2026-01-19 at 20:53 +0100, Jakub Sitnicki wrote:
> Prepare to remove support for calling kfuncs from prologue & epilogue.
> 
> Instead allow direct helpers calls using BPF_EMIT_CALL. Such calls already
> contain helper offset relative to __bpf_call_base and must bypass the
> verifier's patch_call_imm fixup, which expects BPF helper IDs rather than a
> pre-resolved offsets.
> 
> Add a finalized_call flag to bpf_insn_aux_data to mark call instructions
> with resolved offsets so the verifier can skip patch_call_imm fixup for
> these calls.
> 
> Note that the target of BPF_EMIT_CALL should be wrapped with BPF_CALL_x to
> prevent an ABI mismatch between BPF and C on 32-bit architectures.
> 
> Suggested-by: Alexei Starovoitov <ast@kernel.org>
> Signed-off-by: Jakub Sitnicki <jakub@cloudflare.com>
> ---

Reviewed-by: Eduard Zingerman <eddyz87@gmail.com>

[...]

> @@ -21867,6 +21880,8 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env)
>  			ret = add_kfunc_in_insns(env, insn_buf, cnt - 1);
>  			if (ret < 0)
>  				return ret;
> +
> +			mark_helper_calls_finalized(env, 0, cnt - 1);

Note to reviewers:
  `cnt - 1` is because each prologue-generating function does
  `*insn++ = prog->insnsi[0];` in the end. Confusing every time.

>  		}
>  	}
>  
> @@ -21880,6 +21895,7 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env)
>  
>  	for (i = 0; i < insn_cnt; i++, insn++) {
>  		bpf_convert_ctx_access_t convert_ctx_access;
> +		bool is_epilogue = false;

Nit: maybe rename this to finalize_helper_calls and untie from epilogue_idx?
     In case someone would want to add a kfunc call not in an epilogue?

>  		u8 mode;
>  
>  		if (env->insn_aux_data[i + delta].nospec) {

[...]

@@ -23477,6 +23497,9 @@ static int do_misc_fixups(struct bpf_verifier_env *env)
 			goto next_insn;
 		}
 patch_call_imm:
+		if (env->insn_aux_data[i + delta].finalized_call)
+			goto next_insn;
+

Note: This jumps over env->ops->get_func_proto() call.
      Which means that env->ops will not have means to specialize
      helper calls inside pro/epilogue. Not a problem at the moment,
      as the only helper called seem to be 'bpf_skb_pull_data' and
      it does not appear to have alternative implementations.
      Something to keep in mind when extending the code, though.

 		fn = env->ops->get_func_proto(insn->imm, env->prog);
 		/* all functions that have prototype and verifier allowed
 		 * programs to call them, must be real in-kernel functions
[...]

Re: [PATCH bpf-next 2/4] bpf: net_sched: Use direct helper calls instead of kfuncs in pro/epilogue

[Eduard Zingerman]

On Mon, 2026-01-19 at 20:53 +0100, Jakub Sitnicki wrote:
> Convert bpf_qdisc prologue and epilogue to use BPF_EMIT_CALL for direct
> helper calls instead of BPF_CALL_KFUNC.
> 
> Remove the BTF_ID_LIST entries for these functions since they are no longer
> registered as kfuncs.
> 
> Signed-off-by: Jakub Sitnicki <jakub@cloudflare.com>
> ---

Reviewed-by: Eduard Zingerman <eddyz87@gmail.com>

[...]

Re: [PATCH bpf-next 3/4] bpf: Remove kfunc support in prologue and epilogue

[Eduard Zingerman]

On Mon, 2026-01-19 at 20:53 +0100, Jakub Sitnicki wrote:
> Remove add_kfunc_in_insns() and its call sites in convert_ctx_accesses().
> This function was used to register kfuncs found in prologue and epilogue
> instructions, but is no longer needed now that we use direct helper calls
> via BPF_EMIT_CALL instead.
> 
> Signed-off-by: Jakub Sitnicki <jakub@cloudflare.com>
> ---

I think that patches #3 and #4 have to be swapped, otherwise there is
a selftest failure when only patches #1-3 are applied:

  #281/17  pro_epilogue/syscall_pro_epilogue:FAIL

If we want to keep selftests passing for arbitrary bisects.

Reviewed-by: Eduard Zingerman <eddyz87@gmail.com>

[...]

Re: [PATCH bpf-next 4/4] selftests/bpf: Remove tests for prologue/epilogue with kfuncs

[Eduard Zingerman]

On Mon, 2026-01-19 at 20:53 +0100, Jakub Sitnicki wrote:
> Remove pro_epilogue_with_kfunc test program and its supporting code in
> bpf_testmod. This test exercised calling kfuncs from prologue and epilogue,
> which is no longer supported after the switch to direct helper calls.
> 
> Signed-off-by: Jakub Sitnicki <jakub@cloudflare.com>
> ---

Reviewed-by: Eduard Zingerman <eddyz87@gmail.com>

[...]

Re: [PATCH bpf-next 1/4] bpf, verifier: Support direct helper calls from prologue/epilogue

[Martin KaFai Lau]

On 1/19/26 11:53 AM, Jakub Sitnicki wrote:
> Prepare to remove support for calling kfuncs from prologue & epilogue.
> 
> Instead allow direct helpers calls using BPF_EMIT_CALL. Such calls already
> contain helper offset relative to __bpf_call_base and must bypass the
> verifier's patch_call_imm fixup, which expects BPF helper IDs rather than a
> pre-resolved offsets.
> 
> Add a finalized_call flag to bpf_insn_aux_data to mark call instructions
> with resolved offsets so the verifier can skip patch_call_imm fixup for
> these calls.
> 
> Note that the target of BPF_EMIT_CALL should be wrapped with BPF_CALL_x to
> prevent an ABI mismatch between BPF and C on 32-bit architectures.

Acked-by: Martin KaFai Lau <martin.lau@kernel.org>

Re: [PATCH bpf-next 2/4] bpf: net_sched: Use direct helper calls instead of kfuncs in pro/epilogue

[Martin KaFai Lau]

On 1/19/26 11:53 AM, Jakub Sitnicki wrote:
> Convert bpf_qdisc prologue and epilogue to use BPF_EMIT_CALL for direct
> helper calls instead of BPF_CALL_KFUNC.
> 
> Remove the BTF_ID_LIST entries for these functions since they are no longer
> registered as kfuncs.

Acked-by: Martin KaFai Lau <martin.lau@kernel.org>

Re: [PATCH bpf-next 4/4] selftests/bpf: Remove tests for prologue/epilogue with kfuncs

[Martin KaFai Lau]

On 1/19/26 11:53 AM, Jakub Sitnicki wrote:
> Remove pro_epilogue_with_kfunc test program and its supporting code in
> bpf_testmod. This test exercised calling kfuncs from prologue and epilogue,
> which is no longer supported after the switch to direct helper calls.

Is it easy to change it to test calling helper in pro/epilogue?

Re: [PATCH bpf-next 4/4] selftests/bpf: Remove tests for prologue/epilogue with kfuncs

[Jakub Sitnicki]

On Tue, Jan 20, 2026 at 03:48 PM -08, Martin KaFai Lau wrote:
> On 1/19/26 11:53 AM, Jakub Sitnicki wrote:
>> Remove pro_epilogue_with_kfunc test program and its supporting code in
>> bpf_testmod. This test exercised calling kfuncs from prologue and epilogue,
>> which is no longer supported after the switch to direct helper calls.
>
> Is it easy to change it to test calling helper in pro/epilogue?

I gave it a shot and the obstacle is that bpf_cgroup_from_id and
bpf_cgroup_release are not exported symbols, so they can't be referred
to from bpf_testmod. We'd have to move the whole thing to lib/test_bpf
so it's a built-in.

Re: [PATCH bpf-next 3/4] bpf: Remove kfunc support in prologue and epilogue

[Jakub Sitnicki]

On Mon, Jan 19, 2026 at 02:55 PM -08, Eduard Zingerman wrote:
> I think that patches #3 and #4 have to be swapped, otherwise there is
> a selftest failure when only patches #1-3 are applied:
>
>   #281/17  pro_epilogue/syscall_pro_epilogue:FAIL
>
> If we want to keep selftests passing for arbitrary bisects.

I don't have an opinion. I assumed it'd get applied as a batch without
fast forward, so bisect would be going over merge commits.

Re: [PATCH bpf-next 3/4] bpf: Remove kfunc support in prologue and epilogue

[Alexei Starovoitov]

On Wed, Jan 21, 2026 at 1:54 AM Jakub Sitnicki <jakub@cloudflare.com> wrote:
>
> On Mon, Jan 19, 2026 at 02:55 PM -08, Eduard Zingerman wrote:
> > I think that patches #3 and #4 have to be swapped, otherwise there is
> > a selftest failure when only patches #1-3 are applied:
> >
> >   #281/17  pro_epilogue/syscall_pro_epilogue:FAIL
> >
> > If we want to keep selftests passing for arbitrary bisects.
>
> I don't have an opinion. I assumed it'd get applied as a batch without
> fast forward, so bisect would be going over merge commits.

I think 'bisect over merge commits' is rarely the case.
It's better to reorder as Eduard suggested just to be safe.

Re: [PATCH bpf-next 4/4] selftests/bpf: Remove tests for prologue/epilogue with kfuncs

[Martin KaFai Lau]

On 1/21/26 1:49 AM, Jakub Sitnicki wrote:
> On Tue, Jan 20, 2026 at 03:48 PM -08, Martin KaFai Lau wrote:
>> On 1/19/26 11:53 AM, Jakub Sitnicki wrote:
>>> Remove pro_epilogue_with_kfunc test program and its supporting code in
>>> bpf_testmod. This test exercised calling kfuncs from prologue and epilogue,
>>> which is no longer supported after the switch to direct helper calls.
>>
>> Is it easy to change it to test calling helper in pro/epilogue?
> 
> I gave it a shot and the obstacle is that bpf_cgroup_from_id and
> bpf_cgroup_release are not exported symbols, so they can't be referred
> to from bpf_testmod. We'd have to move the whole thing to lib/test_bpf
> so it's a built-in.

I was thinking of creating a helper in bpf_testmod.c but then noticed it 
may fail in the JIT that needs bpf_jit_supports_far_kfunc_call(). A 
cleaner way could be to use the existing pro/epilogue in bpf_qdisc ops. 
I was wondering if the __xlated test here could be quickly adapted in 
patch 4 but it seems that's not the case. The existing bpf_qdisc is 
testing it quietly without the __xlated, so no need to hold up this set.

For patch 3 and 4,

Acked-by: Martin KaFai Lau <martin.lau@kernel.org>

Re: [PATCH bpf-next 0/4] Switch from kfuncs to direct helper calls in prologue/epilogue

[Amery Hung]

On Mon, Jan 19, 2026 at 11:53 AM Jakub Sitnicki <jakub@cloudflare.com> wrote:
>
> This series enables direct helper calls using BPF_EMIT_CALL from prologue
> and epilogue code generated by verifier ops. The goal is to simplify the
> calling convention and remove kfunc support from prologue/epilogue, as
> suggested by Alexei [1].
>
> Patch 1 adds the infrastructure to mark direct helper calls as finalized
> (already resolved) so the verifier skips the imm fixup.
>
> Patch 2 converts bpf_qdisc to use BPF_EMIT_CALL instead of BPF_CALL_KFUNC
> for the init prologue and reset/destroy epilogue helpers.
>
> Patch 3 removes the now-unused kfunc support code from prologue/epilogue
> handling in the verifier.
>
> Patch 4 removes the corresponding selftests that exercised kfuncs in
> prologue/epilogue.
>

Thanks for converting bpf qdisc to using BPF_EMIT_CALL. The patchset
looks good to me.

Reviewed-by: Amery Hung <ameryhung@gmail.com>

> [1] https://lore.kernel.org/bpf/CAADnVQJ=kmVAZsgkG9P2nEBTUG3E4PrDG=Yz8tfeFysH4ZBqVw@mail.gmail.com/
>
> Signed-off-by: Jakub Sitnicki <jakub@cloudflare.com>
> ---
> Jakub Sitnicki (4):
>       bpf, verifier: Support direct helper calls from prologue/epilogue
>       bpf: net_sched: Use direct helper calls instead of kfuncs in pro/epilogue
>       bpf: Remove kfunc support in prologue and epilogue
>       selftests/bpf: Remove tests for prologue/epilogue with kfuncs
>
>  include/linux/bpf_verifier.h                       |  1 +
>  kernel/bpf/verifier.c                              | 47 +++++------
>  net/core/filter.c                                  |  3 +-
>  net/sched/bpf_qdisc.c                              | 76 ++++++++----------
>  .../selftests/bpf/prog_tests/pro_epilogue.c        |  2 -
>  .../selftests/bpf/progs/pro_epilogue_with_kfunc.c  | 88 ---------------------
>  .../testing/selftests/bpf/test_kmods/bpf_testmod.c | 92 ----------------------
>  7 files changed, 61 insertions(+), 248 deletions(-)
>