[bpf-next v8 0/5] emit ENDBR/BTI instructions for indirect jump targets

public inbox for linux-arm-kernel@lists.infradead.org
 help / color / mirror / Atom feed

* [bpf-next v8 0/5] emit ENDBR/BTI instructions for indirect jump targets
@ 2026-03-09 14:00 Xu Kuohai
  2026-03-09 14:00 ` [bpf-next v8 1/5] bpf: Move constants blinding from JIT to verifier Xu Kuohai
                   ` (6 more replies)
  0 siblings, 7 replies; 20+ messages in thread
From: Xu Kuohai @ 2026-03-09 14:00 UTC (permalink / raw)
  To: bpf, linux-kernel, linux-arm-kernel
  Cc: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
	Martin KaFai Lau, Eduard Zingerman, Yonghong Song, Puranjay Mohan,
	Anton Protopopov, Shahab Vahedi, Russell King, Tiezhu Yang,
	Hengqi Chen, Johan Almbladh, Paul Burton, Hari Bathini,
	Christophe Leroy, Naveen N Rao, Luke Nelson, Xi Wang,
	Björn Töpel, Pu Lehui, Ilya Leoshkevich, Heiko Carstens,
	Vasily Gorbik, David S . Miller, Wang YanQing

On architectures with CFI protection enabled that require landing pad
instructions at indirect jump targets, such as x86 with CET/IBT eanbled
and arm64 with BTI enabled, kernel panics when an indirect jump lands on
a target witout landing pad. Therefore, the JIT must emit landing pad
instructions for indirect jump targets.

The verifier already recognizes which instructions are indirect jump
targets during the verification phase. So we can stores this information
in env->insn_aux_data and pass it to the JIT as new parameter, so the JIT
knows which instructions are indirect jump targets.

During JIT, constants blinding is performed. It rewrites the private copy
of instructions for the JITed program, but it does not adjust the global
env->insn_aux_data array. As a result, after constants blinding, the
instruction indexes used by JIT may no longer match the indexes in
env->insn_aux_data, so the JIT can not lookup env->insn_aux_data directly.

To avoid this mistach, and considering that all existing arch-specific JITs
already implement constants blinding with largely duplicated code, move
constants blinding from JIT to generic code, before copying instructions
for each subprog.

v8:
- Define void bpf_jit_blind_constants() function when CONFIG_BPF_JIT is not set 
- Move indirect_target fixup for insn patching from bpf_jit_blind_constants()
  to adjust_insn_aux_data()

v7: https://lore.kernel.org/bpf/20260307103949.2340104-1-xukuohai@huaweicloud.com
- Move constants blinding logic back to bpf/core.c
- Compute ip address before switch statement in x86 JIT
- Clear JIT state from error path on arm64 and loongarch 

v6: https://lore.kernel.org/bpf/20260306102329.2056216-1-xukuohai@huaweicloud.com/
- Move constants blinding from JIT to verifier
- Move call to bpf_prog_select_runtime from bpf_prog_load to verifier

v5: https://lore.kernel.org/bpf/20260302102726.1126019-1-xukuohai@huaweicloud.com/
- Switch to pass env to JIT directly to get rid of coping private insn_aux_data for
  each prog

v4: https://lore.kernel.org/all/20260114093914.2403982-1-xukuohai@huaweicloud.com/
- Switch to the approach proposed by Eduard, using insn_aux_data to indentify indirect
  jump targets, and emit ENDBR on x86

v3: https://lore.kernel.org/bpf/20251227081033.240336-1-xukuohai@huaweicloud.com/
- Get rid of unnecessary enum definition (Yonghong Song, Anton Protopopov)

v2: https://lore.kernel.org/bpf/20251223085447.139301-1-xukuohai@huaweicloud.com/
- Exclude instruction arrays not used for indirect jumps (Anton Protopopov)

v1: https://lore.kernel.org/bpf/20251127140318.3944249-1-xukuohai@huaweicloud.com/

Xu Kuohai (5):
  bpf: Move constants blinding from JIT to verifier
  bpf: Pass bpf_verifier_env to JIT
  bpf: Add helper to detect indirect jump targets
  bpf, x86: Emit ENDBR for indirect jump targets
  bpf, arm64: Emit BTI for indirect jump target

 arch/arc/net/bpf_jit_core.c      |  37 +++-----
 arch/arm/net/bpf_jit_32.c        |  43 ++--------
 arch/arm64/net/bpf_jit_comp.c    |  86 +++++++------------
 arch/loongarch/net/bpf_jit.c     |  58 ++++---------
 arch/mips/net/bpf_jit_comp.c     |  22 +----
 arch/parisc/net/bpf_jit_core.c   |  40 ++-------
 arch/powerpc/net/bpf_jit_comp.c  |  47 +++-------
 arch/riscv/net/bpf_jit_core.c    |  47 +++-------
 arch/s390/net/bpf_jit_comp.c     |  43 ++--------
 arch/sparc/net/bpf_jit_comp_64.c |  43 ++--------
 arch/x86/net/bpf_jit_comp.c      |  68 +++++----------
 arch/x86/net/bpf_jit_comp32.c    |  35 ++------
 include/linux/bpf.h              |   2 +
 include/linux/bpf_verifier.h     |   9 +-
 include/linux/filter.h           |  15 +++-
 kernel/bpf/core.c                | 142 +++++++++----------------------
 kernel/bpf/syscall.c             |   4 -
 kernel/bpf/verifier.c            |  45 +++++++---
 18 files changed, 233 insertions(+), 553 deletions(-)

-- 
2.47.3



^ permalink raw reply	[flat|nested] 20+ messages in thread

* [bpf-next v8 1/5] bpf: Move constants blinding from JIT to verifier
  2026-03-09 14:00 [bpf-next v8 0/5] emit ENDBR/BTI instructions for indirect jump targets Xu Kuohai
@ 2026-03-09 14:00 ` Xu Kuohai
  2026-03-09 17:20   ` Anton Protopopov
                     ` (2 more replies)
  2026-03-09 14:00 ` [bpf-next v8 2/5] bpf: Pass bpf_verifier_env to JIT Xu Kuohai
                   ` (5 subsequent siblings)
  6 siblings, 3 replies; 20+ messages in thread
From: Xu Kuohai @ 2026-03-09 14:00 UTC (permalink / raw)
  To: bpf, linux-kernel, linux-arm-kernel
  Cc: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
	Martin KaFai Lau, Eduard Zingerman, Yonghong Song, Puranjay Mohan,
	Anton Protopopov, Shahab Vahedi, Russell King, Tiezhu Yang,
	Hengqi Chen, Johan Almbladh, Paul Burton, Hari Bathini,
	Christophe Leroy, Naveen N Rao, Luke Nelson, Xi Wang,
	Björn Töpel, Pu Lehui, Ilya Leoshkevich, Heiko Carstens,
	Vasily Gorbik, David S . Miller, Wang YanQing

From: Xu Kuohai <xukuohai@huawei.com>

During the JIT stage, constants blinding rewrites instructions but only
rewrites the private instruction copy of the JITed subprog, leaving the
global instructions and insn_aux_data unchanged. This causes a mismatch
between subprog instructions and the global state, making it difficult
to look up the global insn_aux_data in the JIT.

To avoid this mismatch, and given that all arch-specific JITs already
support constants blinding, move it to the generic verifier code, and
switch to rewrite the global env->insnsi with the global states
adjusted, as other rewrites in the verifier do.

This removes the constant blinding calls in each JIT, which are largely
duplicated code across architectures.

And the prog clone functions and insn_array adjustment for the JIT
constant blinding are no longer needed, remove them too.

Signed-off-by: Xu Kuohai <xukuohai@huawei.com>
---
 arch/arc/net/bpf_jit_core.c      | 20 +------
 arch/arm/net/bpf_jit_32.c        | 41 +++----------
 arch/arm64/net/bpf_jit_comp.c    | 71 +++++++----------------
 arch/loongarch/net/bpf_jit.c     | 56 +++++-------------
 arch/mips/net/bpf_jit_comp.c     | 20 +------
 arch/parisc/net/bpf_jit_core.c   | 38 +++---------
 arch/powerpc/net/bpf_jit_comp.c  | 45 ++++-----------
 arch/riscv/net/bpf_jit_core.c    | 45 ++++-----------
 arch/s390/net/bpf_jit_comp.c     | 41 +++----------
 arch/sparc/net/bpf_jit_comp_64.c | 41 +++----------
 arch/x86/net/bpf_jit_comp.c      | 40 ++-----------
 arch/x86/net/bpf_jit_comp32.c    | 33 ++---------
 include/linux/filter.h           | 11 +++-
 kernel/bpf/core.c                | 99 +++++---------------------------
 kernel/bpf/verifier.c            | 19 +++---
 15 files changed, 127 insertions(+), 493 deletions(-)

diff --git a/arch/arc/net/bpf_jit_core.c b/arch/arc/net/bpf_jit_core.c
index 1421eeced0f5..12facf5750da 100644
--- a/arch/arc/net/bpf_jit_core.c
+++ b/arch/arc/net/bpf_jit_core.c
@@ -79,7 +79,6 @@ struct arc_jit_data {
  * The JIT pertinent context that is used by different functions.
  *
  * prog:		The current eBPF program being handled.
- * orig_prog:		The original eBPF program before any possible change.
  * jit:			The JIT buffer and its length.
  * bpf_header:		The JITed program header. "jit.buf" points inside it.
  * emit:		If set, opcodes are written to memory; else, a dry-run.
@@ -94,12 +93,10 @@ struct arc_jit_data {
  * need_extra_pass:	A forecast if an "extra_pass" will occur.
  * is_extra_pass:	Indicates if the current pass is an extra pass.
  * user_bpf_prog:	True, if VM opcodes come from a real program.
- * blinded:		True if "constant blinding" step returned a new "prog".
  * success:		Indicates if the whole JIT went OK.
  */
 struct jit_context {
 	struct bpf_prog			*prog;
-	struct bpf_prog			*orig_prog;
 	struct jit_buffer		jit;
 	struct bpf_binary_header	*bpf_header;
 	bool				emit;
@@ -114,7 +111,6 @@ struct jit_context {
 	bool				need_extra_pass;
 	bool				is_extra_pass;
 	bool				user_bpf_prog;
-	bool				blinded;
 	bool				success;
 };
 
@@ -161,13 +157,7 @@ static int jit_ctx_init(struct jit_context *ctx, struct bpf_prog *prog)
 {
 	memset(ctx, 0, sizeof(*ctx));
 
-	ctx->orig_prog = prog;
-
-	/* If constant blinding was requested but failed, scram. */
-	ctx->prog = bpf_jit_blind_constants(prog);
-	if (IS_ERR(ctx->prog))
-		return PTR_ERR(ctx->prog);
-	ctx->blinded = (ctx->prog != ctx->orig_prog);
+	ctx->prog = prog;
 
 	/* If the verifier doesn't zero-extend, then we have to do it. */
 	ctx->do_zext = !ctx->prog->aux->verifier_zext;
@@ -214,14 +204,6 @@ static inline void maybe_free(struct jit_context *ctx, void **mem)
  */
 static void jit_ctx_cleanup(struct jit_context *ctx)
 {
-	if (ctx->blinded) {
-		/* if all went well, release the orig_prog. */
-		if (ctx->success)
-			bpf_jit_prog_release_other(ctx->prog, ctx->orig_prog);
-		else
-			bpf_jit_prog_release_other(ctx->orig_prog, ctx->prog);
-	}
-
 	maybe_free(ctx, (void **)&ctx->bpf2insn);
 	maybe_free(ctx, (void **)&ctx->jit_data);
 
diff --git a/arch/arm/net/bpf_jit_32.c b/arch/arm/net/bpf_jit_32.c
index deeb8f292454..e6b1bb2de627 100644
--- a/arch/arm/net/bpf_jit_32.c
+++ b/arch/arm/net/bpf_jit_32.c
@@ -2144,9 +2144,7 @@ bool bpf_jit_needs_zext(void)
 
 struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 {
-	struct bpf_prog *tmp, *orig_prog = prog;
 	struct bpf_binary_header *header;
-	bool tmp_blinded = false;
 	struct jit_ctx ctx;
 	unsigned int tmp_idx;
 	unsigned int image_size;
@@ -2156,20 +2154,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 	 * the interpreter.
 	 */
 	if (!prog->jit_requested)
-		return orig_prog;
-
-	/* If constant blinding was enabled and we failed during blinding
-	 * then we must fall back to the interpreter. Otherwise, we save
-	 * the new JITed code.
-	 */
-	tmp = bpf_jit_blind_constants(prog);
-
-	if (IS_ERR(tmp))
-		return orig_prog;
-	if (tmp != prog) {
-		tmp_blinded = true;
-		prog = tmp;
-	}
+		return prog;
 
 	memset(&ctx, 0, sizeof(ctx));
 	ctx.prog = prog;
@@ -2179,10 +2164,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 	 * we must fall back to the interpreter
 	 */
 	ctx.offsets = kcalloc(prog->len, sizeof(int), GFP_KERNEL);
-	if (ctx.offsets == NULL) {
-		prog = orig_prog;
-		goto out;
-	}
+	if (ctx.offsets == NULL)
+		return prog;
 
 	/* 1) fake pass to find in the length of the JITed code,
 	 * to compute ctx->offsets and other context variables
@@ -2194,10 +2177,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 	 * being successful in the second pass, so just fall back
 	 * to the interpreter.
 	 */
-	if (build_body(&ctx)) {
-		prog = orig_prog;
+	if (build_body(&ctx))
 		goto out_off;
-	}
 
 	tmp_idx = ctx.idx;
 	build_prologue(&ctx);
@@ -2213,10 +2194,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 	ctx.idx += ctx.imm_count;
 	if (ctx.imm_count) {
 		ctx.imms = kcalloc(ctx.imm_count, sizeof(u32), GFP_KERNEL);
-		if (ctx.imms == NULL) {
-			prog = orig_prog;
+		if (ctx.imms == NULL)
 			goto out_off;
-		}
 	}
 #else
 	/* there's nothing about the epilogue on ARMv7 */
@@ -2238,10 +2217,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 	/* Not able to allocate memory for the structure then
 	 * we must fall back to the interpretation
 	 */
-	if (header == NULL) {
-		prog = orig_prog;
+	if (header == NULL)
 		goto out_imms;
-	}
 
 	/* 2.) Actual pass to generate final JIT code */
 	ctx.target = (u32 *) image_ptr;
@@ -2278,16 +2255,12 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 #endif
 out_off:
 	kfree(ctx.offsets);
-out:
-	if (tmp_blinded)
-		bpf_jit_prog_release_other(prog, prog == orig_prog ?
-					   tmp : orig_prog);
+
 	return prog;
 
 out_free:
 	image_ptr = NULL;
 	bpf_jit_binary_free(header);
-	prog = orig_prog;
 	goto out_imms;
 }
 
diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
index adf84962d579..566809be4a02 100644
--- a/arch/arm64/net/bpf_jit_comp.c
+++ b/arch/arm64/net/bpf_jit_comp.c
@@ -2006,17 +2006,22 @@ struct arm64_jit_data {
 	struct jit_ctx ctx;
 };
 
+static void clear_jit_state(struct bpf_prog *prog)
+{
+	prog->bpf_func = NULL;
+	prog->jited = 0;
+	prog->jited_len = 0;
+}
+
 struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 {
 	int image_size, prog_size, extable_size, extable_align, extable_offset;
-	struct bpf_prog *tmp, *orig_prog = prog;
 	struct bpf_binary_header *header;
 	struct bpf_binary_header *ro_header = NULL;
 	struct arm64_jit_data *jit_data;
 	void __percpu *priv_stack_ptr = NULL;
 	bool was_classic = bpf_prog_was_classic(prog);
 	int priv_stack_alloc_sz;
-	bool tmp_blinded = false;
 	bool extra_pass = false;
 	struct jit_ctx ctx;
 	u8 *image_ptr;
@@ -2025,26 +2030,13 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 	int exentry_idx;
 
 	if (!prog->jit_requested)
-		return orig_prog;
-
-	tmp = bpf_jit_blind_constants(prog);
-	/* If blinding was requested and we failed during blinding,
-	 * we must fall back to the interpreter.
-	 */
-	if (IS_ERR(tmp))
-		return orig_prog;
-	if (tmp != prog) {
-		tmp_blinded = true;
-		prog = tmp;
-	}
+		return prog;
 
 	jit_data = prog->aux->jit_data;
 	if (!jit_data) {
 		jit_data = kzalloc_obj(*jit_data);
-		if (!jit_data) {
-			prog = orig_prog;
-			goto out;
-		}
+		if (!jit_data)
+			return prog;
 		prog->aux->jit_data = jit_data;
 	}
 	priv_stack_ptr = prog->aux->priv_stack_ptr;
@@ -2056,10 +2048,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 		priv_stack_alloc_sz = round_up(prog->aux->stack_depth, 16) +
 				      2 * PRIV_STACK_GUARD_SZ;
 		priv_stack_ptr = __alloc_percpu_gfp(priv_stack_alloc_sz, 16, GFP_KERNEL);
-		if (!priv_stack_ptr) {
-			prog = orig_prog;
+		if (!priv_stack_ptr)
 			goto out_priv_stack;
-		}
 
 		priv_stack_init_guard(priv_stack_ptr, priv_stack_alloc_sz);
 		prog->aux->priv_stack_ptr = priv_stack_ptr;
@@ -2079,10 +2069,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 	ctx.prog = prog;
 
 	ctx.offset = kvzalloc_objs(int, prog->len + 1);
-	if (ctx.offset == NULL) {
-		prog = orig_prog;
+	if (ctx.offset == NULL)
 		goto out_off;
-	}
 
 	ctx.user_vm_start = bpf_arena_get_user_vm_start(prog->aux->arena);
 	ctx.arena_vm_start = bpf_arena_get_kern_vm_start(prog->aux->arena);
@@ -2095,15 +2083,11 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 	 * BPF line info needs ctx->offset[i] to be the offset of
 	 * instruction[i] in jited image, so build prologue first.
 	 */
-	if (build_prologue(&ctx, was_classic)) {
-		prog = orig_prog;
+	if (build_prologue(&ctx, was_classic))
 		goto out_off;
-	}
 
-	if (build_body(&ctx, extra_pass)) {
-		prog = orig_prog;
+	if (build_body(&ctx, extra_pass))
 		goto out_off;
-	}
 
 	ctx.epilogue_offset = ctx.idx;
 	build_epilogue(&ctx, was_classic);
@@ -2121,10 +2105,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 	ro_header = bpf_jit_binary_pack_alloc(image_size, &ro_image_ptr,
 					      sizeof(u64), &header, &image_ptr,
 					      jit_fill_hole);
-	if (!ro_header) {
-		prog = orig_prog;
+	if (!ro_header)
 		goto out_off;
-	}
 
 	/* Pass 2: Determine jited position and result for each instruction */
 
@@ -2152,10 +2134,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 	/* Dont write body instructions to memory for now */
 	ctx.write = false;
 
-	if (build_body(&ctx, extra_pass)) {
-		prog = orig_prog;
+	if (build_body(&ctx, extra_pass))
 		goto out_free_hdr;
-	}
 
 	ctx.epilogue_offset = ctx.idx;
 	ctx.exentry_idx = exentry_idx;
@@ -2164,19 +2144,15 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 
 	/* Pass 3: Adjust jump offset and write final image */
 	if (build_body(&ctx, extra_pass) ||
-		WARN_ON_ONCE(ctx.idx != ctx.epilogue_offset)) {
-		prog = orig_prog;
+		WARN_ON_ONCE(ctx.idx != ctx.epilogue_offset))
 		goto out_free_hdr;
-	}
 
 	build_epilogue(&ctx, was_classic);
 	build_plt(&ctx);
 
 	/* Extra pass to validate JITed code. */
-	if (validate_ctx(&ctx)) {
-		prog = orig_prog;
+	if (validate_ctx(&ctx))
 		goto out_free_hdr;
-	}
 
 	/* update the real prog size */
 	prog_size = sizeof(u32) * ctx.idx;
@@ -2193,15 +2169,13 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 		if (extra_pass && ctx.idx > jit_data->ctx.idx) {
 			pr_err_once("multi-func JIT bug %d > %d\n",
 				    ctx.idx, jit_data->ctx.idx);
-			prog->bpf_func = NULL;
-			prog->jited = 0;
-			prog->jited_len = 0;
+			clear_jit_state(prog);
 			goto out_free_hdr;
 		}
 		if (WARN_ON(bpf_jit_binary_pack_finalize(ro_header, header))) {
 			/* ro_header has been freed */
 			ro_header = NULL;
-			prog = orig_prog;
+			clear_jit_state(prog);
 			goto out_off;
 		}
 		/*
@@ -2245,10 +2219,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 		kfree(jit_data);
 		prog->aux->jit_data = NULL;
 	}
-out:
-	if (tmp_blinded)
-		bpf_jit_prog_release_other(prog, prog == orig_prog ?
-					   tmp : orig_prog);
+
 	return prog;
 
 out_free_hdr:
diff --git a/arch/loongarch/net/bpf_jit.c b/arch/loongarch/net/bpf_jit.c
index 3bd89f55960d..57dd24d53c77 100644
--- a/arch/loongarch/net/bpf_jit.c
+++ b/arch/loongarch/net/bpf_jit.c
@@ -1911,43 +1911,26 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
 
 struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 {
-	bool tmp_blinded = false, extra_pass = false;
+	bool extra_pass = false;
 	u8 *image_ptr, *ro_image_ptr;
 	int image_size, prog_size, extable_size;
 	struct jit_ctx ctx;
 	struct jit_data *jit_data;
 	struct bpf_binary_header *header;
 	struct bpf_binary_header *ro_header;
-	struct bpf_prog *tmp, *orig_prog = prog;
 
 	/*
 	 * If BPF JIT was not enabled then we must fall back to
 	 * the interpreter.
 	 */
 	if (!prog->jit_requested)
-		return orig_prog;
-
-	tmp = bpf_jit_blind_constants(prog);
-	/*
-	 * If blinding was requested and we failed during blinding,
-	 * we must fall back to the interpreter. Otherwise, we save
-	 * the new JITed code.
-	 */
-	if (IS_ERR(tmp))
-		return orig_prog;
-
-	if (tmp != prog) {
-		tmp_blinded = true;
-		prog = tmp;
-	}
+		return prog;
 
 	jit_data = prog->aux->jit_data;
 	if (!jit_data) {
 		jit_data = kzalloc_obj(*jit_data);
-		if (!jit_data) {
-			prog = orig_prog;
-			goto out;
-		}
+		if (!jit_data)
+			return prog;
 		prog->aux->jit_data = jit_data;
 	}
 	if (jit_data->ctx.offset) {
@@ -1967,17 +1950,13 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 	ctx.user_vm_start = bpf_arena_get_user_vm_start(prog->aux->arena);
 
 	ctx.offset = kvcalloc(prog->len + 1, sizeof(u32), GFP_KERNEL);
-	if (ctx.offset == NULL) {
-		prog = orig_prog;
+	if (ctx.offset == NULL)
 		goto out_offset;
-	}
 
 	/* 1. Initial fake pass to compute ctx->idx and set ctx->flags */
 	build_prologue(&ctx);
-	if (build_body(&ctx, extra_pass)) {
-		prog = orig_prog;
+	if (build_body(&ctx, extra_pass))
 		goto out_offset;
-	}
 	ctx.epilogue_offset = ctx.idx;
 	build_epilogue(&ctx);
 
@@ -1993,10 +1972,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 	/* Now we know the size of the structure to make */
 	ro_header = bpf_jit_binary_pack_alloc(image_size, &ro_image_ptr, sizeof(u32),
 					      &header, &image_ptr, jit_fill_hole);
-	if (!ro_header) {
-		prog = orig_prog;
+	if (!ro_header)
 		goto out_offset;
-	}
 
 	/* 2. Now, the actual pass to generate final JIT code */
 	/*
@@ -2016,17 +1993,13 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 	ctx.num_exentries = 0;
 
 	build_prologue(&ctx);
-	if (build_body(&ctx, extra_pass)) {
-		prog = orig_prog;
+	if (build_body(&ctx, extra_pass))
 		goto out_free;
-	}
 	build_epilogue(&ctx);
 
 	/* 3. Extra pass to validate JITed code */
-	if (validate_ctx(&ctx)) {
-		prog = orig_prog;
+	if (validate_ctx(&ctx))
 		goto out_free;
-	}
 
 	/* And we're done */
 	if (bpf_jit_enable > 1)
@@ -2041,7 +2014,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 		if (WARN_ON(bpf_jit_binary_pack_finalize(ro_header, header))) {
 			/* ro_header has been freed */
 			ro_header = NULL;
-			prog = orig_prog;
 			goto out_free;
 		}
 		/*
@@ -2073,13 +2045,15 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 		prog->aux->jit_data = NULL;
 	}
 
-out:
-	if (tmp_blinded)
-		bpf_jit_prog_release_other(prog, prog == orig_prog ? tmp : orig_prog);
-
 	return prog;
 
 out_free:
+	if (prog->jited) {
+		prog->bpf_func = NULL;
+		prog->jited = 0;
+		prog->jited_len = 0;
+	}
+
 	if (header) {
 		bpf_arch_text_copy(&ro_header->size, &header->size, sizeof(header->size));
 		bpf_jit_binary_pack_free(ro_header, header);
diff --git a/arch/mips/net/bpf_jit_comp.c b/arch/mips/net/bpf_jit_comp.c
index e355dfca4400..d2b6c955f18e 100644
--- a/arch/mips/net/bpf_jit_comp.c
+++ b/arch/mips/net/bpf_jit_comp.c
@@ -911,10 +911,8 @@ bool bpf_jit_needs_zext(void)
 
 struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 {
-	struct bpf_prog *tmp, *orig_prog = prog;
 	struct bpf_binary_header *header = NULL;
 	struct jit_context ctx;
-	bool tmp_blinded = false;
 	unsigned int tmp_idx;
 	unsigned int image_size;
 	u8 *image_ptr;
@@ -925,19 +923,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 	 * the interpreter.
 	 */
 	if (!prog->jit_requested)
-		return orig_prog;
-	/*
-	 * If constant blinding was enabled and we failed during blinding
-	 * then we must fall back to the interpreter. Otherwise, we save
-	 * the new JITed code.
-	 */
-	tmp = bpf_jit_blind_constants(prog);
-	if (IS_ERR(tmp))
-		return orig_prog;
-	if (tmp != prog) {
-		tmp_blinded = true;
-		prog = tmp;
-	}
+		return prog;
 
 	memset(&ctx, 0, sizeof(ctx));
 	ctx.program = prog;
@@ -1025,14 +1011,10 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 	prog->jited_len = image_size;
 
 out:
-	if (tmp_blinded)
-		bpf_jit_prog_release_other(prog, prog == orig_prog ?
-					   tmp : orig_prog);
 	kfree(ctx.descriptors);
 	return prog;
 
 out_err:
-	prog = orig_prog;
 	if (header)
 		bpf_jit_binary_free(header);
 	goto out;
diff --git a/arch/parisc/net/bpf_jit_core.c b/arch/parisc/net/bpf_jit_core.c
index a5eb6b51e27a..4d339636a34a 100644
--- a/arch/parisc/net/bpf_jit_core.c
+++ b/arch/parisc/net/bpf_jit_core.c
@@ -44,30 +44,19 @@ bool bpf_jit_needs_zext(void)
 struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 {
 	unsigned int prog_size = 0, extable_size = 0;
-	bool tmp_blinded = false, extra_pass = false;
-	struct bpf_prog *tmp, *orig_prog = prog;
+	bool extra_pass = false;
 	int pass = 0, prev_ninsns = 0, prologue_len, i;
 	struct hppa_jit_data *jit_data;
 	struct hppa_jit_context *ctx;
 
 	if (!prog->jit_requested)
-		return orig_prog;
-
-	tmp = bpf_jit_blind_constants(prog);
-	if (IS_ERR(tmp))
-		return orig_prog;
-	if (tmp != prog) {
-		tmp_blinded = true;
-		prog = tmp;
-	}
+		return prog;
 
 	jit_data = prog->aux->jit_data;
 	if (!jit_data) {
 		jit_data = kzalloc_obj(*jit_data);
-		if (!jit_data) {
-			prog = orig_prog;
-			goto out;
-		}
+		if (!jit_data)
+			return prog;
 		prog->aux->jit_data = jit_data;
 	}
 
@@ -81,10 +70,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 
 	ctx->prog = prog;
 	ctx->offset = kzalloc_objs(int, prog->len);
-	if (!ctx->offset) {
-		prog = orig_prog;
+	if (!ctx->offset)
 		goto out_offset;
-	}
 	for (i = 0; i < prog->len; i++) {
 		prev_ninsns += 20;
 		ctx->offset[i] = prev_ninsns;
@@ -93,10 +80,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 	for (i = 0; i < NR_JIT_ITERATIONS; i++) {
 		pass++;
 		ctx->ninsns = 0;
-		if (build_body(ctx, extra_pass, ctx->offset)) {
-			prog = orig_prog;
+		if (build_body(ctx, extra_pass, ctx->offset))
 			goto out_offset;
-		}
 		ctx->body_len = ctx->ninsns;
 		bpf_jit_build_prologue(ctx);
 		ctx->prologue_len = ctx->ninsns - ctx->body_len;
@@ -116,10 +101,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 						     &jit_data->image,
 						     sizeof(long),
 						     bpf_fill_ill_insns);
-			if (!jit_data->header) {
-				prog = orig_prog;
+			if (!jit_data->header)
 				goto out_offset;
-			}
 
 			ctx->insns = (u32 *)jit_data->image;
 			/*
@@ -134,7 +117,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 		pr_err("bpf-jit: image did not converge in <%d passes!\n", i);
 		if (jit_data->header)
 			bpf_jit_binary_free(jit_data->header);
-		prog = orig_prog;
 		goto out_offset;
 	}
 
@@ -148,7 +130,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 	bpf_jit_build_prologue(ctx);
 	if (build_body(ctx, extra_pass, NULL)) {
 		bpf_jit_binary_free(jit_data->header);
-		prog = orig_prog;
 		goto out_offset;
 	}
 	bpf_jit_build_epilogue(ctx);
@@ -183,13 +164,10 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 		kfree(jit_data);
 		prog->aux->jit_data = NULL;
 	}
-out:
+
 	if (HPPA_JIT_REBOOT)
 		{ extern int machine_restart(char *); machine_restart(""); }
 
-	if (tmp_blinded)
-		bpf_jit_prog_release_other(prog, prog == orig_prog ?
-					   tmp : orig_prog);
 	return prog;
 }
 
diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_comp.c
index 52162e4a7f84..7a7c49640a2f 100644
--- a/arch/powerpc/net/bpf_jit_comp.c
+++ b/arch/powerpc/net/bpf_jit_comp.c
@@ -142,9 +142,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
 	int flen;
 	struct bpf_binary_header *fhdr = NULL;
 	struct bpf_binary_header *hdr = NULL;
-	struct bpf_prog *org_fp = fp;
-	struct bpf_prog *tmp_fp;
-	bool bpf_blinded = false;
 	bool extra_pass = false;
 	u8 *fimage = NULL;
 	u32 *fcode_base;
@@ -152,24 +149,13 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
 	u32 fixup_len;
 
 	if (!fp->jit_requested)
-		return org_fp;
-
-	tmp_fp = bpf_jit_blind_constants(org_fp);
-	if (IS_ERR(tmp_fp))
-		return org_fp;
-
-	if (tmp_fp != org_fp) {
-		bpf_blinded = true;
-		fp = tmp_fp;
-	}
+		return fp;
 
 	jit_data = fp->aux->jit_data;
 	if (!jit_data) {
 		jit_data = kzalloc_obj(*jit_data);
-		if (!jit_data) {
-			fp = org_fp;
-			goto out;
-		}
+		if (!jit_data)
+			return fp;
 		fp->aux->jit_data = jit_data;
 	}
 
@@ -194,10 +180,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
 	}
 
 	addrs = kcalloc(flen + 1, sizeof(*addrs), GFP_KERNEL);
-	if (addrs == NULL) {
-		fp = org_fp;
+	if (addrs == NULL)
 		goto out_addrs;
-	}
 
 	memset(&cgctx, 0, sizeof(struct codegen_context));
 	bpf_jit_init_reg_mapping(&cgctx);
@@ -211,11 +195,9 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
 	cgctx.exception_cb = fp->aux->exception_cb;
 
 	/* Scouting faux-generate pass 0 */
-	if (bpf_jit_build_body(fp, NULL, NULL, &cgctx, addrs, 0, false)) {
+	if (bpf_jit_build_body(fp, NULL, NULL, &cgctx, addrs, 0, false))
 		/* We hit something illegal or unsupported. */
-		fp = org_fp;
 		goto out_addrs;
-	}
 
 	/*
 	 * If we have seen a tail call, we need a second pass.
@@ -226,10 +208,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
 	 */
 	if (cgctx.seen & SEEN_TAILCALL || !is_offset_in_branch_range((long)cgctx.idx * 4)) {
 		cgctx.idx = 0;
-		if (bpf_jit_build_body(fp, NULL, NULL, &cgctx, addrs, 0, false)) {
-			fp = org_fp;
+		if (bpf_jit_build_body(fp, NULL, NULL, &cgctx, addrs, 0, false))
 			goto out_addrs;
-		}
 	}
 
 	bpf_jit_realloc_regs(&cgctx);
@@ -250,10 +230,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
 
 	fhdr = bpf_jit_binary_pack_alloc(alloclen, &fimage, 4, &hdr, &image,
 					      bpf_jit_fill_ill_insns);
-	if (!fhdr) {
-		fp = org_fp;
+	if (!fhdr)
 		goto out_addrs;
-	}
 
 	if (extable_len)
 		fp->aux->extable = (void *)fimage + FUNCTION_DESCR_SIZE + proglen + fixup_len;
@@ -272,7 +250,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
 				       extra_pass)) {
 			bpf_arch_text_copy(&fhdr->size, &hdr->size, sizeof(hdr->size));
 			bpf_jit_binary_pack_free(fhdr, hdr);
-			fp = org_fp;
 			goto out_addrs;
 		}
 		bpf_jit_build_epilogue(code_base, &cgctx);
@@ -301,7 +278,9 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
 
 	if (!fp->is_func || extra_pass) {
 		if (bpf_jit_binary_pack_finalize(fhdr, hdr)) {
-			fp = org_fp;
+			fp->bpf_func = NULL;
+			fp->jited = 0;
+			fp->jited_len = 0;
 			goto out_addrs;
 		}
 		bpf_prog_fill_jited_linfo(fp, addrs);
@@ -318,10 +297,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
 		jit_data->hdr = hdr;
 	}
 
-out:
-	if (bpf_blinded)
-		bpf_jit_prog_release_other(fp, fp == org_fp ? tmp_fp : org_fp);
-
 	return fp;
 }
 
diff --git a/arch/riscv/net/bpf_jit_core.c b/arch/riscv/net/bpf_jit_core.c
index b3581e926436..c77e8aba14d3 100644
--- a/arch/riscv/net/bpf_jit_core.c
+++ b/arch/riscv/net/bpf_jit_core.c
@@ -44,29 +44,19 @@ bool bpf_jit_needs_zext(void)
 struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 {
 	unsigned int prog_size = 0, extable_size = 0;
-	bool tmp_blinded = false, extra_pass = false;
-	struct bpf_prog *tmp, *orig_prog = prog;
+	bool extra_pass = false;
 	int pass = 0, prev_ninsns = 0, i;
 	struct rv_jit_data *jit_data;
 	struct rv_jit_context *ctx;
 
 	if (!prog->jit_requested)
-		return orig_prog;
-
-	tmp = bpf_jit_blind_constants(prog);
-	if (IS_ERR(tmp))
-		return orig_prog;
-	if (tmp != prog) {
-		tmp_blinded = true;
-		prog = tmp;
-	}
+		return prog;
 
 	jit_data = prog->aux->jit_data;
 	if (!jit_data) {
 		jit_data = kzalloc_obj(*jit_data);
 		if (!jit_data) {
-			prog = orig_prog;
-			goto out;
+			return prog;
 		}
 		prog->aux->jit_data = jit_data;
 	}
@@ -83,15 +73,11 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 	ctx->user_vm_start = bpf_arena_get_user_vm_start(prog->aux->arena);
 	ctx->prog = prog;
 	ctx->offset = kzalloc_objs(int, prog->len);
-	if (!ctx->offset) {
-		prog = orig_prog;
+	if (!ctx->offset)
 		goto out_offset;
-	}
 
-	if (build_body(ctx, extra_pass, NULL)) {
-		prog = orig_prog;
+	if (build_body(ctx, extra_pass, NULL))
 		goto out_offset;
-	}
 
 	for (i = 0; i < prog->len; i++) {
 		prev_ninsns += 32;
@@ -105,10 +91,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 		bpf_jit_build_prologue(ctx, bpf_is_subprog(prog));
 		ctx->prologue_len = ctx->ninsns;
 
-		if (build_body(ctx, extra_pass, ctx->offset)) {
-			prog = orig_prog;
+		if (build_body(ctx, extra_pass, ctx->offset))
 			goto out_offset;
-		}
 
 		ctx->epilogue_offset = ctx->ninsns;
 		bpf_jit_build_epilogue(ctx);
@@ -126,10 +110,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 							  &jit_data->ro_image, sizeof(u32),
 							  &jit_data->header, &jit_data->image,
 							  bpf_fill_ill_insns);
-			if (!jit_data->ro_header) {
-				prog = orig_prog;
+			if (!jit_data->ro_header)
 				goto out_offset;
-			}
 
 			/*
 			 * Use the image(RW) for writing the JITed instructions. But also save
@@ -150,7 +132,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 
 	if (i == NR_JIT_ITERATIONS) {
 		pr_err("bpf-jit: image did not converge in <%d passes!\n", i);
-		prog = orig_prog;
 		goto out_free_hdr;
 	}
 
@@ -163,10 +144,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 	ctx->nexentries = 0;
 
 	bpf_jit_build_prologue(ctx, bpf_is_subprog(prog));
-	if (build_body(ctx, extra_pass, NULL)) {
-		prog = orig_prog;
+	if (build_body(ctx, extra_pass, NULL))
 		goto out_free_hdr;
-	}
 	bpf_jit_build_epilogue(ctx);
 
 	if (bpf_jit_enable > 1)
@@ -180,7 +159,9 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 		if (WARN_ON(bpf_jit_binary_pack_finalize(jit_data->ro_header, jit_data->header))) {
 			/* ro_header has been freed */
 			jit_data->ro_header = NULL;
-			prog = orig_prog;
+			prog->bpf_func = NULL;
+			prog->jited = 0;
+			prog->jited_len = 0;
 			goto out_offset;
 		}
 		/*
@@ -198,11 +179,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 		kfree(jit_data);
 		prog->aux->jit_data = NULL;
 	}
-out:
 
-	if (tmp_blinded)
-		bpf_jit_prog_release_other(prog, prog == orig_prog ?
-					   tmp : orig_prog);
 	return prog;
 
 out_free_hdr:
diff --git a/arch/s390/net/bpf_jit_comp.c b/arch/s390/net/bpf_jit_comp.c
index 1f9a6b728beb..d6de2abfe4a7 100644
--- a/arch/s390/net/bpf_jit_comp.c
+++ b/arch/s390/net/bpf_jit_comp.c
@@ -2305,36 +2305,20 @@ static struct bpf_binary_header *bpf_jit_alloc(struct bpf_jit *jit,
  */
 struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
 {
-	struct bpf_prog *tmp, *orig_fp = fp;
 	struct bpf_binary_header *header;
 	struct s390_jit_data *jit_data;
-	bool tmp_blinded = false;
 	bool extra_pass = false;
 	struct bpf_jit jit;
 	int pass;
 
 	if (!fp->jit_requested)
-		return orig_fp;
-
-	tmp = bpf_jit_blind_constants(fp);
-	/*
-	 * If blinding was requested and we failed during blinding,
-	 * we must fall back to the interpreter.
-	 */
-	if (IS_ERR(tmp))
-		return orig_fp;
-	if (tmp != fp) {
-		tmp_blinded = true;
-		fp = tmp;
-	}
+		return fp;
 
 	jit_data = fp->aux->jit_data;
 	if (!jit_data) {
 		jit_data = kzalloc_obj(*jit_data);
-		if (!jit_data) {
-			fp = orig_fp;
-			goto out;
-		}
+		if (!jit_data)
+			return fp;
 		fp->aux->jit_data = jit_data;
 	}
 	if (jit_data->ctx.addrs) {
@@ -2347,33 +2331,26 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
 
 	memset(&jit, 0, sizeof(jit));
 	jit.addrs = kvcalloc(fp->len + 1, sizeof(*jit.addrs), GFP_KERNEL);
-	if (jit.addrs == NULL) {
-		fp = orig_fp;
+	if (jit.addrs == NULL)
 		goto free_addrs;
-	}
 	/*
 	 * Three initial passes:
 	 *   - 1/2: Determine clobbered registers
 	 *   - 3:   Calculate program size and addrs array
 	 */
 	for (pass = 1; pass <= 3; pass++) {
-		if (bpf_jit_prog(&jit, fp, extra_pass)) {
-			fp = orig_fp;
+		if (bpf_jit_prog(&jit, fp, extra_pass))
 			goto free_addrs;
-		}
 	}
 	/*
 	 * Final pass: Allocate and generate program
 	 */
 	header = bpf_jit_alloc(&jit, fp);
-	if (!header) {
-		fp = orig_fp;
+	if (!header)
 		goto free_addrs;
-	}
 skip_init_ctx:
 	if (bpf_jit_prog(&jit, fp, extra_pass)) {
 		bpf_jit_binary_free(header);
-		fp = orig_fp;
 		goto free_addrs;
 	}
 	if (bpf_jit_enable > 1) {
@@ -2383,7 +2360,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
 	if (!fp->is_func || extra_pass) {
 		if (bpf_jit_binary_lock_ro(header)) {
 			bpf_jit_binary_free(header);
-			fp = orig_fp;
 			goto free_addrs;
 		}
 	} else {
@@ -2402,10 +2378,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
 		kfree(jit_data);
 		fp->aux->jit_data = NULL;
 	}
-out:
-	if (tmp_blinded)
-		bpf_jit_prog_release_other(fp, fp == orig_fp ?
-					   tmp : orig_fp);
+
 	return fp;
 }
 
diff --git a/arch/sparc/net/bpf_jit_comp_64.c b/arch/sparc/net/bpf_jit_comp_64.c
index b23d1c645ae5..86abd84d4005 100644
--- a/arch/sparc/net/bpf_jit_comp_64.c
+++ b/arch/sparc/net/bpf_jit_comp_64.c
@@ -1479,37 +1479,22 @@ struct sparc64_jit_data {
 
 struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 {
-	struct bpf_prog *tmp, *orig_prog = prog;
 	struct sparc64_jit_data *jit_data;
 	struct bpf_binary_header *header;
 	u32 prev_image_size, image_size;
-	bool tmp_blinded = false;
 	bool extra_pass = false;
 	struct jit_ctx ctx;
 	u8 *image_ptr;
 	int pass, i;
 
 	if (!prog->jit_requested)
-		return orig_prog;
-
-	tmp = bpf_jit_blind_constants(prog);
-	/* If blinding was requested and we failed during blinding,
-	 * we must fall back to the interpreter.
-	 */
-	if (IS_ERR(tmp))
-		return orig_prog;
-	if (tmp != prog) {
-		tmp_blinded = true;
-		prog = tmp;
-	}
+		return prog;
 
 	jit_data = prog->aux->jit_data;
 	if (!jit_data) {
 		jit_data = kzalloc_obj(*jit_data);
-		if (!jit_data) {
-			prog = orig_prog;
-			goto out;
-		}
+		if (!jit_data)
+			return prog;
 		prog->aux->jit_data = jit_data;
 	}
 	if (jit_data->ctx.offset) {
@@ -1527,10 +1512,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 	ctx.prog = prog;
 
 	ctx.offset = kmalloc_array(prog->len, sizeof(unsigned int), GFP_KERNEL);
-	if (ctx.offset == NULL) {
-		prog = orig_prog;
+	if (ctx.offset == NULL)
 		goto out_off;
-	}
 
 	/* Longest sequence emitted is for bswap32, 12 instructions.  Pre-cook
 	 * the offset array so that we converge faster.
@@ -1543,10 +1526,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 		ctx.idx = 0;
 
 		build_prologue(&ctx);
-		if (build_body(&ctx)) {
-			prog = orig_prog;
+		if (build_body(&ctx))
 			goto out_off;
-		}
 		build_epilogue(&ctx);
 
 		if (bpf_jit_enable > 1)
@@ -1569,10 +1550,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 	image_size = sizeof(u32) * ctx.idx;
 	header = bpf_jit_binary_alloc(image_size, &image_ptr,
 				      sizeof(u32), jit_fill_hole);
-	if (header == NULL) {
-		prog = orig_prog;
+	if (header == NULL)
 		goto out_off;
-	}
 
 	ctx.image = (u32 *)image_ptr;
 skip_init_ctx:
@@ -1582,7 +1561,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 
 	if (build_body(&ctx)) {
 		bpf_jit_binary_free(header);
-		prog = orig_prog;
 		goto out_off;
 	}
 
@@ -1592,7 +1570,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 		pr_err("bpf_jit: Failed to converge, prev_size=%u size=%d\n",
 		       prev_image_size, ctx.idx * 4);
 		bpf_jit_binary_free(header);
-		prog = orig_prog;
 		goto out_off;
 	}
 
@@ -1604,7 +1581,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 	if (!prog->is_func || extra_pass) {
 		if (bpf_jit_binary_lock_ro(header)) {
 			bpf_jit_binary_free(header);
-			prog = orig_prog;
 			goto out_off;
 		}
 	} else {
@@ -1624,9 +1600,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 		kfree(jit_data);
 		prog->aux->jit_data = NULL;
 	}
-out:
-	if (tmp_blinded)
-		bpf_jit_prog_release_other(prog, prog == orig_prog ?
-					   tmp : orig_prog);
+
 	return prog;
 }
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index e9b78040d703..de51ab3a11ee 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -3717,13 +3717,11 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 {
 	struct bpf_binary_header *rw_header = NULL;
 	struct bpf_binary_header *header = NULL;
-	struct bpf_prog *tmp, *orig_prog = prog;
 	void __percpu *priv_stack_ptr = NULL;
 	struct x64_jit_data *jit_data;
 	int priv_stack_alloc_sz;
 	int proglen, oldproglen = 0;
 	struct jit_context ctx = {};
-	bool tmp_blinded = false;
 	bool extra_pass = false;
 	bool padding = false;
 	u8 *rw_image = NULL;
@@ -3733,27 +3731,13 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 	int i;
 
 	if (!prog->jit_requested)
-		return orig_prog;
-
-	tmp = bpf_jit_blind_constants(prog);
-	/*
-	 * If blinding was requested and we failed during blinding,
-	 * we must fall back to the interpreter.
-	 */
-	if (IS_ERR(tmp))
-		return orig_prog;
-	if (tmp != prog) {
-		tmp_blinded = true;
-		prog = tmp;
-	}
+		return prog;
 
 	jit_data = prog->aux->jit_data;
 	if (!jit_data) {
 		jit_data = kzalloc_obj(*jit_data);
-		if (!jit_data) {
-			prog = orig_prog;
+		if (!jit_data)
 			goto out;
-		}
 		prog->aux->jit_data = jit_data;
 	}
 	priv_stack_ptr = prog->aux->priv_stack_ptr;
@@ -3765,10 +3749,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 		priv_stack_alloc_sz = round_up(prog->aux->stack_depth, 8) +
 				      2 * PRIV_STACK_GUARD_SZ;
 		priv_stack_ptr = __alloc_percpu_gfp(priv_stack_alloc_sz, 8, GFP_KERNEL);
-		if (!priv_stack_ptr) {
-			prog = orig_prog;
+		if (!priv_stack_ptr)
 			goto out_priv_stack;
-		}
 
 		priv_stack_init_guard(priv_stack_ptr, priv_stack_alloc_sz);
 		prog->aux->priv_stack_ptr = priv_stack_ptr;
@@ -3786,10 +3768,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 		goto skip_init_addrs;
 	}
 	addrs = kvmalloc_objs(*addrs, prog->len + 1);
-	if (!addrs) {
-		prog = orig_prog;
+	if (!addrs)
 		goto out_addrs;
-	}
 
 	/*
 	 * Before first pass, make a rough estimation of addrs[]
@@ -3820,8 +3800,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 						   sizeof(rw_header->size));
 				bpf_jit_binary_pack_free(header, rw_header);
 			}
-			/* Fall back to interpreter mode */
-			prog = orig_prog;
 			if (extra_pass) {
 				prog->bpf_func = NULL;
 				prog->jited = 0;
@@ -3852,10 +3830,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 			header = bpf_jit_binary_pack_alloc(roundup(proglen, align) + extable_size,
 							   &image, align, &rw_header, &rw_image,
 							   jit_fill_hole);
-			if (!header) {
-				prog = orig_prog;
+			if (!header)
 				goto out_addrs;
-			}
 			prog->aux->extable = (void *) image + roundup(proglen, align);
 		}
 		oldproglen = proglen;
@@ -3908,8 +3884,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 		prog->bpf_func = (void *)image + cfi_get_offset();
 		prog->jited = 1;
 		prog->jited_len = proglen - cfi_get_offset();
-	} else {
-		prog = orig_prog;
 	}
 
 	if (!image || !prog->is_func || extra_pass) {
@@ -3925,10 +3899,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 		kfree(jit_data);
 		prog->aux->jit_data = NULL;
 	}
+
 out:
-	if (tmp_blinded)
-		bpf_jit_prog_release_other(prog, prog == orig_prog ?
-					   tmp : orig_prog);
 	return prog;
 }
 
diff --git a/arch/x86/net/bpf_jit_comp32.c b/arch/x86/net/bpf_jit_comp32.c
index dda423025c3d..5f259577614a 100644
--- a/arch/x86/net/bpf_jit_comp32.c
+++ b/arch/x86/net/bpf_jit_comp32.c
@@ -2521,35 +2521,19 @@ bool bpf_jit_needs_zext(void)
 struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 {
 	struct bpf_binary_header *header = NULL;
-	struct bpf_prog *tmp, *orig_prog = prog;
 	int proglen, oldproglen = 0;
 	struct jit_context ctx = {};
-	bool tmp_blinded = false;
 	u8 *image = NULL;
 	int *addrs;
 	int pass;
 	int i;
 
 	if (!prog->jit_requested)
-		return orig_prog;
-
-	tmp = bpf_jit_blind_constants(prog);
-	/*
-	 * If blinding was requested and we failed during blinding,
-	 * we must fall back to the interpreter.
-	 */
-	if (IS_ERR(tmp))
-		return orig_prog;
-	if (tmp != prog) {
-		tmp_blinded = true;
-		prog = tmp;
-	}
+		return prog;
 
 	addrs = kmalloc_objs(*addrs, prog->len);
-	if (!addrs) {
-		prog = orig_prog;
-		goto out;
-	}
+	if (!addrs)
+		return prog;
 
 	/*
 	 * Before first pass, make a rough estimation of addrs[]
@@ -2574,7 +2558,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 			image = NULL;
 			if (header)
 				bpf_jit_binary_free(header);
-			prog = orig_prog;
 			goto out_addrs;
 		}
 		if (image) {
@@ -2588,10 +2571,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 		if (proglen == oldproglen) {
 			header = bpf_jit_binary_alloc(proglen, &image,
 						      1, jit_fill_hole);
-			if (!header) {
-				prog = orig_prog;
+			if (!header)
 				goto out_addrs;
-			}
 		}
 		oldproglen = proglen;
 		cond_resched();
@@ -2604,16 +2585,10 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 		prog->bpf_func = (void *)image;
 		prog->jited = 1;
 		prog->jited_len = proglen;
-	} else {
-		prog = orig_prog;
 	}
 
 out_addrs:
 	kfree(addrs);
-out:
-	if (tmp_blinded)
-		bpf_jit_prog_release_other(prog, prog == orig_prog ?
-					   tmp : orig_prog);
 	return prog;
 }
 
diff --git a/include/linux/filter.h b/include/linux/filter.h
index 44d7ae95ddbc..2484d85be63d 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -1184,6 +1184,10 @@ static inline bool bpf_dump_raw_ok(const struct cred *cred)
 
 struct bpf_prog *bpf_patch_insn_single(struct bpf_prog *prog, u32 off,
 				       const struct bpf_insn *patch, u32 len);
+
+struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, u32 off,
+				     const struct bpf_insn *patch, u32 len);
+
 int bpf_remove_insns(struct bpf_prog *prog, u32 off, u32 cnt);
 
 static inline bool xdp_return_frame_no_direct(void)
@@ -1310,8 +1314,7 @@ int bpf_jit_get_func_addr(const struct bpf_prog *prog,
 
 const char *bpf_jit_get_prog_name(struct bpf_prog *prog);
 
-struct bpf_prog *bpf_jit_blind_constants(struct bpf_prog *fp);
-void bpf_jit_prog_release_other(struct bpf_prog *fp, struct bpf_prog *fp_other);
+int bpf_jit_blind_constants(struct bpf_verifier_env *env);
 
 static inline void bpf_jit_dump(unsigned int flen, unsigned int proglen,
 				u32 pass, void *image)
@@ -1451,6 +1454,10 @@ static inline void bpf_prog_kallsyms_del(struct bpf_prog *fp)
 {
 }
 
+static inline int bpf_jit_blind_constants(struct bpf_verifier_env *env)
+{
+	return 0;
+}
 #endif /* CONFIG_BPF_JIT */
 
 void bpf_prog_kallsyms_del_all(struct bpf_prog *fp);
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 229c74f3d6ae..c692213b1fdf 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -1427,82 +1427,19 @@ static int bpf_jit_blind_insn(const struct bpf_insn *from,
 	return to - to_buff;
 }
 
-static struct bpf_prog *bpf_prog_clone_create(struct bpf_prog *fp_other,
-					      gfp_t gfp_extra_flags)
-{
-	gfp_t gfp_flags = GFP_KERNEL | __GFP_ZERO | gfp_extra_flags;
-	struct bpf_prog *fp;
-
-	fp = __vmalloc(fp_other->pages * PAGE_SIZE, gfp_flags);
-	if (fp != NULL) {
-		/* aux->prog still points to the fp_other one, so
-		 * when promoting the clone to the real program,
-		 * this still needs to be adapted.
-		 */
-		memcpy(fp, fp_other, fp_other->pages * PAGE_SIZE);
-	}
-
-	return fp;
-}
-
-static void bpf_prog_clone_free(struct bpf_prog *fp)
-{
-	/* aux was stolen by the other clone, so we cannot free
-	 * it from this path! It will be freed eventually by the
-	 * other program on release.
-	 *
-	 * At this point, we don't need a deferred release since
-	 * clone is guaranteed to not be locked.
-	 */
-	fp->aux = NULL;
-	fp->stats = NULL;
-	fp->active = NULL;
-	__bpf_prog_free(fp);
-}
-
-void bpf_jit_prog_release_other(struct bpf_prog *fp, struct bpf_prog *fp_other)
-{
-	/* We have to repoint aux->prog to self, as we don't
-	 * know whether fp here is the clone or the original.
-	 */
-	fp->aux->prog = fp;
-	bpf_prog_clone_free(fp_other);
-}
-
-static void adjust_insn_arrays(struct bpf_prog *prog, u32 off, u32 len)
-{
-#ifdef CONFIG_BPF_SYSCALL
-	struct bpf_map *map;
-	int i;
-
-	if (len <= 1)
-		return;
-
-	for (i = 0; i < prog->aux->used_map_cnt; i++) {
-		map = prog->aux->used_maps[i];
-		if (map->map_type == BPF_MAP_TYPE_INSN_ARRAY)
-			bpf_insn_array_adjust(map, off, len);
-	}
-#endif
-}
-
-struct bpf_prog *bpf_jit_blind_constants(struct bpf_prog *prog)
+int bpf_jit_blind_constants(struct bpf_verifier_env *env)
 {
 	struct bpf_insn insn_buff[16], aux[2];
-	struct bpf_prog *clone, *tmp;
+	struct bpf_prog *prog = env->prog;
 	int insn_delta, insn_cnt;
 	struct bpf_insn *insn;
 	int i, rewritten;
 
 	if (!prog->blinding_requested || prog->blinded)
-		return prog;
-
-	clone = bpf_prog_clone_create(prog, GFP_USER);
-	if (!clone)
-		return ERR_PTR(-ENOMEM);
+		return 0;
 
-	insn_cnt = clone->len;
-	insn = clone->insnsi;
+	insn_cnt = prog->len;
+	insn = prog->insnsi;
 
 	for (i = 0; i < insn_cnt; i++, insn++) {
 		if (bpf_pseudo_func(insn)) {
@@ -1523,35 +1460,25 @@ struct bpf_prog *bpf_jit_blind_constants(struct bpf_prog *prog)
 		    insn[1].code == 0)
 			memcpy(aux, insn, sizeof(aux));
 
-		rewritten = bpf_jit_blind_insn(insn, aux, insn_buff,
-						clone->aux->verifier_zext);
+		rewritten = bpf_jit_blind_insn(insn, aux, insn_buff, prog->aux->verifier_zext);
 		if (!rewritten)
 			continue;
 
-		tmp = bpf_patch_insn_single(clone, i, insn_buff, rewritten);
-		if (IS_ERR(tmp)) {
-			/* Patching may have repointed aux->prog during
-			 * realloc from the original one, so we need to
-			 * fix it up here on error.
-			 */
-			bpf_jit_prog_release_other(prog, clone);
-			return tmp;
-		}
+		prog = bpf_patch_insn_data(env, i, insn_buff, rewritten);
+		if (!prog)
+			return -ENOMEM;
 
-		clone = tmp;
+		env->prog = prog;
 		insn_delta = rewritten - 1;
 
-		/* Instructions arrays must be updated using absolute xlated offsets */
-		adjust_insn_arrays(clone, prog->aux->subprog_start + i, rewritten);
-
 		/* Walk new program and skip insns we just inserted. */
-		insn = clone->insnsi + i + insn_delta;
+		insn = prog->insnsi + i + insn_delta;
 		insn_cnt += insn_delta;
 		i        += insn_delta;
 	}
 
-	clone->blinded = 1;
-	return clone;
+	prog->blinded = 1;
+	return 0;
 }
 #endif /* CONFIG_BPF_JIT */
 
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 7aa06f534cb2..e290c9b7d13d 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -22070,8 +22070,8 @@ static void adjust_poke_descs(struct bpf_prog *prog, u32 off, u32 len)
 	}
 }
 
-static struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, u32 off,
-					    const struct bpf_insn *patch, u32 len)
+struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, u32 off,
+				     const struct bpf_insn *patch, u32 len)
 {
 	struct bpf_prog *new_prog;
 	struct bpf_insn_aux_data *new_data = NULL;
@@ -22846,7 +22846,6 @@ static int jit_subprogs(struct bpf_verifier_env *env)
 	struct bpf_insn *insn;
 	void *old_bpf_func;
 	int err, num_exentries;
-	int old_len, subprog_start_adjustment = 0;
 
 	if (env->subprog_cnt <= 1)
 		return 0;
@@ -22918,10 +22917,11 @@ static int jit_subprogs(struct bpf_verifier_env *env)
 			goto out_free;
 		func[i]->is_func = 1;
 		func[i]->sleepable = prog->sleepable;
+		func[i]->blinded = prog->blinded;
 		func[i]->aux->func_idx = i;
 		/* Below members will be freed only at prog->aux */
 		func[i]->aux->btf = prog->aux->btf;
-		func[i]->aux->subprog_start = subprog_start + subprog_start_adjustment;
+		func[i]->aux->subprog_start = subprog_start;
 		func[i]->aux->func_info = prog->aux->func_info;
 		func[i]->aux->func_info_cnt = prog->aux->func_info_cnt;
 		func[i]->aux->poke_tab = prog->aux->poke_tab;
@@ -22977,15 +22977,7 @@ static int jit_subprogs(struct bpf_verifier_env *env)
 		func[i]->aux->might_sleep = env->subprog_info[i].might_sleep;
 		if (!i)
 			func[i]->aux->exception_boundary = env->seen_exception;
-
-		/*
-		 * To properly pass the absolute subprog start to jit
-		 * all instruction adjustments should be accumulated
-		 */
-		old_len = func[i]->len;
 		func[i] = bpf_int_jit_compile(func[i]);
-		subprog_start_adjustment += func[i]->len - old_len;
-
 		if (!func[i]->jited) {
 			err = -ENOTSUPP;
 			goto out_free;
@@ -23136,6 +23128,9 @@ static int fixup_call_args(struct bpf_verifier_env *env)
 
 	if (env->prog->jit_requested &&
 	    !bpf_prog_is_offloaded(env->prog->aux)) {
+		err = bpf_jit_blind_constants(env);
+		if (err)
+			return err;
 		err = jit_subprogs(env);
 		if (err == 0)
 			return 0;
-- 
2.47.3



^ permalink raw reply related	[flat|nested] 20+ messages in thread

* Re: [bpf-next v8 1/5] bpf: Move constants blinding from JIT to verifier
  2026-03-09 14:00 ` [bpf-next v8 1/5] bpf: Move constants blinding from JIT to verifier Xu Kuohai
@ 2026-03-09 17:20   ` Anton Protopopov
  2026-03-10  6:52     ` Xu Kuohai
  2026-03-09 21:25   ` Eduard Zingerman
  2026-03-17 10:55   ` kernel test robot
  2 siblings, 1 reply; 20+ messages in thread
From: Anton Protopopov @ 2026-03-09 17:20 UTC (permalink / raw)
  To: Xu Kuohai
  Cc: bpf, linux-kernel, linux-arm-kernel, Alexei Starovoitov,
	Daniel Borkmann, Andrii Nakryiko, Martin KaFai Lau,
	Eduard Zingerman, Yonghong Song, Puranjay Mohan, Shahab Vahedi,
	Russell King, Tiezhu Yang, Hengqi Chen, Johan Almbladh,
	Paul Burton, Hari Bathini, Christophe Leroy, Naveen N Rao,
	Luke Nelson, Xi Wang, Björn Töpel, Pu Lehui,
	Ilya Leoshkevich, Heiko Carstens, Vasily Gorbik, David S . Miller,
	Wang YanQing

On 26/03/09 10:00PM, Xu Kuohai wrote:
> From: Xu Kuohai <xukuohai@huawei.com>
> 
> During the JIT stage, constants blinding rewrites instructions but only
> rewrites the private instruction copy of the JITed subprog, leaving the
> global instructions and insn_aux_data unchanged. This causes a mismatch
> between subprog instructions and the global state, making it difficult
> to look up the global insn_aux_data in the JIT.
> 
> To avoid this mismatch, and given that all arch-specific JITs already
> support constants blinding, move it to the generic verifier code, and
> switch to rewrite the global env->insnsi with the global states
> adjusted, as other rewrites in the verifier do.
> 
> This removes the constant blinding calls in each JIT, which are largely
> duplicated code across architectures.
> 
> And the prog clone functions and insn_array adjustment for the JIT
> constant blinding are no longer needed, remove them too.
> 
> Signed-off-by: Xu Kuohai <xukuohai@huawei.com>
> ---
>  arch/arc/net/bpf_jit_core.c      | 20 +------
>  arch/arm/net/bpf_jit_32.c        | 41 +++----------
>  arch/arm64/net/bpf_jit_comp.c    | 71 +++++++----------------
>  arch/loongarch/net/bpf_jit.c     | 56 +++++-------------
>  arch/mips/net/bpf_jit_comp.c     | 20 +------
>  arch/parisc/net/bpf_jit_core.c   | 38 +++---------
>  arch/powerpc/net/bpf_jit_comp.c  | 45 ++++-----------
>  arch/riscv/net/bpf_jit_core.c    | 45 ++++-----------
>  arch/s390/net/bpf_jit_comp.c     | 41 +++----------
>  arch/sparc/net/bpf_jit_comp_64.c | 41 +++----------
>  arch/x86/net/bpf_jit_comp.c      | 40 ++-----------
>  arch/x86/net/bpf_jit_comp32.c    | 33 ++---------
>  include/linux/filter.h           | 11 +++-
>  kernel/bpf/core.c                | 99 +++++---------------------------
>  kernel/bpf/verifier.c            | 19 +++---
>  15 files changed, 127 insertions(+), 493 deletions(-)
> 
> diff --git a/arch/arc/net/bpf_jit_core.c b/arch/arc/net/bpf_jit_core.c
> index 1421eeced0f5..12facf5750da 100644
> --- a/arch/arc/net/bpf_jit_core.c
> +++ b/arch/arc/net/bpf_jit_core.c
> @@ -79,7 +79,6 @@ struct arc_jit_data {
>   * The JIT pertinent context that is used by different functions.
>   *
>   * prog:		The current eBPF program being handled.
> - * orig_prog:		The original eBPF program before any possible change.
>   * jit:			The JIT buffer and its length.
>   * bpf_header:		The JITed program header. "jit.buf" points inside it.
>   * emit:		If set, opcodes are written to memory; else, a dry-run.
> @@ -94,12 +93,10 @@ struct arc_jit_data {
>   * need_extra_pass:	A forecast if an "extra_pass" will occur.
>   * is_extra_pass:	Indicates if the current pass is an extra pass.
>   * user_bpf_prog:	True, if VM opcodes come from a real program.
> - * blinded:		True if "constant blinding" step returned a new "prog".
>   * success:		Indicates if the whole JIT went OK.
>   */
>  struct jit_context {
>  	struct bpf_prog			*prog;
> -	struct bpf_prog			*orig_prog;
>  	struct jit_buffer		jit;
>  	struct bpf_binary_header	*bpf_header;
>  	bool				emit;
> @@ -114,7 +111,6 @@ struct jit_context {
>  	bool				need_extra_pass;
>  	bool				is_extra_pass;
>  	bool				user_bpf_prog;
> -	bool				blinded;
>  	bool				success;
>  };
>  
> @@ -161,13 +157,7 @@ static int jit_ctx_init(struct jit_context *ctx, struct bpf_prog *prog)
>  {
>  	memset(ctx, 0, sizeof(*ctx));
>  
> -	ctx->orig_prog = prog;
> -
> -	/* If constant blinding was requested but failed, scram. */
> -	ctx->prog = bpf_jit_blind_constants(prog);
> -	if (IS_ERR(ctx->prog))
> -		return PTR_ERR(ctx->prog);
> -	ctx->blinded = (ctx->prog != ctx->orig_prog);
> +	ctx->prog = prog;
>  
>  	/* If the verifier doesn't zero-extend, then we have to do it. */
>  	ctx->do_zext = !ctx->prog->aux->verifier_zext;
> @@ -214,14 +204,6 @@ static inline void maybe_free(struct jit_context *ctx, void **mem)
>   */
>  static void jit_ctx_cleanup(struct jit_context *ctx)
>  {
> -	if (ctx->blinded) {
> -		/* if all went well, release the orig_prog. */
> -		if (ctx->success)
> -			bpf_jit_prog_release_other(ctx->prog, ctx->orig_prog);
> -		else
> -			bpf_jit_prog_release_other(ctx->orig_prog, ctx->prog);
> -	}
> -
>  	maybe_free(ctx, (void **)&ctx->bpf2insn);
>  	maybe_free(ctx, (void **)&ctx->jit_data);
>  
> diff --git a/arch/arm/net/bpf_jit_32.c b/arch/arm/net/bpf_jit_32.c
> index deeb8f292454..e6b1bb2de627 100644
> --- a/arch/arm/net/bpf_jit_32.c
> +++ b/arch/arm/net/bpf_jit_32.c
> @@ -2144,9 +2144,7 @@ bool bpf_jit_needs_zext(void)
>  
>  struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  {
> -	struct bpf_prog *tmp, *orig_prog = prog;
>  	struct bpf_binary_header *header;
> -	bool tmp_blinded = false;
>  	struct jit_ctx ctx;
>  	unsigned int tmp_idx;
>  	unsigned int image_size;
> @@ -2156,20 +2154,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  	 * the interpreter.
>  	 */
>  	if (!prog->jit_requested)
> -		return orig_prog;
> -
> -	/* If constant blinding was enabled and we failed during blinding
> -	 * then we must fall back to the interpreter. Otherwise, we save
> -	 * the new JITed code.
> -	 */
> -	tmp = bpf_jit_blind_constants(prog);
> -
> -	if (IS_ERR(tmp))
> -		return orig_prog;
> -	if (tmp != prog) {
> -		tmp_blinded = true;
> -		prog = tmp;
> -	}
> +		return prog;
>  
>  	memset(&ctx, 0, sizeof(ctx));
>  	ctx.prog = prog;
> @@ -2179,10 +2164,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  	 * we must fall back to the interpreter
>  	 */
>  	ctx.offsets = kcalloc(prog->len, sizeof(int), GFP_KERNEL);
> -	if (ctx.offsets == NULL) {
> -		prog = orig_prog;
> -		goto out;
> -	}
> +	if (ctx.offsets == NULL)
> +		return prog;
>  
>  	/* 1) fake pass to find in the length of the JITed code,
>  	 * to compute ctx->offsets and other context variables
> @@ -2194,10 +2177,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  	 * being successful in the second pass, so just fall back
>  	 * to the interpreter.
>  	 */
> -	if (build_body(&ctx)) {
> -		prog = orig_prog;
> +	if (build_body(&ctx))
>  		goto out_off;
> -	}
>  
>  	tmp_idx = ctx.idx;
>  	build_prologue(&ctx);
> @@ -2213,10 +2194,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  	ctx.idx += ctx.imm_count;
>  	if (ctx.imm_count) {
>  		ctx.imms = kcalloc(ctx.imm_count, sizeof(u32), GFP_KERNEL);
> -		if (ctx.imms == NULL) {
> -			prog = orig_prog;
> +		if (ctx.imms == NULL)
>  			goto out_off;
> -		}
>  	}
>  #else
>  	/* there's nothing about the epilogue on ARMv7 */
> @@ -2238,10 +2217,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  	/* Not able to allocate memory for the structure then
>  	 * we must fall back to the interpretation
>  	 */
> -	if (header == NULL) {
> -		prog = orig_prog;
> +	if (header == NULL)
>  		goto out_imms;
> -	}
>  
>  	/* 2.) Actual pass to generate final JIT code */
>  	ctx.target = (u32 *) image_ptr;
> @@ -2278,16 +2255,12 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  #endif
>  out_off:
>  	kfree(ctx.offsets);
> -out:
> -	if (tmp_blinded)
> -		bpf_jit_prog_release_other(prog, prog == orig_prog ?
> -					   tmp : orig_prog);
> +
>  	return prog;
>  
>  out_free:
>  	image_ptr = NULL;
>  	bpf_jit_binary_free(header);
> -	prog = orig_prog;
>  	goto out_imms;
>  }
>  
> diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
> index adf84962d579..566809be4a02 100644
> --- a/arch/arm64/net/bpf_jit_comp.c
> +++ b/arch/arm64/net/bpf_jit_comp.c
> @@ -2006,17 +2006,22 @@ struct arm64_jit_data {
>  	struct jit_ctx ctx;
>  };
>  
> +static void clear_jit_state(struct bpf_prog *prog)
> +{
> +	prog->bpf_func = NULL;
> +	prog->jited = 0;
> +	prog->jited_len = 0;
> +}
> +
>  struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  {
>  	int image_size, prog_size, extable_size, extable_align, extable_offset;
> -	struct bpf_prog *tmp, *orig_prog = prog;
>  	struct bpf_binary_header *header;
>  	struct bpf_binary_header *ro_header = NULL;
>  	struct arm64_jit_data *jit_data;
>  	void __percpu *priv_stack_ptr = NULL;
>  	bool was_classic = bpf_prog_was_classic(prog);
>  	int priv_stack_alloc_sz;
> -	bool tmp_blinded = false;
>  	bool extra_pass = false;
>  	struct jit_ctx ctx;
>  	u8 *image_ptr;
> @@ -2025,26 +2030,13 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  	int exentry_idx;
>  
>  	if (!prog->jit_requested)
> -		return orig_prog;
> -
> -	tmp = bpf_jit_blind_constants(prog);
> -	/* If blinding was requested and we failed during blinding,
> -	 * we must fall back to the interpreter.
> -	 */
> -	if (IS_ERR(tmp))
> -		return orig_prog;
> -	if (tmp != prog) {
> -		tmp_blinded = true;
> -		prog = tmp;
> -	}
> +		return prog;
>  
>  	jit_data = prog->aux->jit_data;
>  	if (!jit_data) {
>  		jit_data = kzalloc_obj(*jit_data);
> -		if (!jit_data) {
> -			prog = orig_prog;
> -			goto out;
> -		}
> +		if (!jit_data)
> +			return prog;
>  		prog->aux->jit_data = jit_data;
>  	}
>  	priv_stack_ptr = prog->aux->priv_stack_ptr;
> @@ -2056,10 +2048,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  		priv_stack_alloc_sz = round_up(prog->aux->stack_depth, 16) +
>  				      2 * PRIV_STACK_GUARD_SZ;
>  		priv_stack_ptr = __alloc_percpu_gfp(priv_stack_alloc_sz, 16, GFP_KERNEL);
> -		if (!priv_stack_ptr) {
> -			prog = orig_prog;
> +		if (!priv_stack_ptr)
>  			goto out_priv_stack;
> -		}
>  
>  		priv_stack_init_guard(priv_stack_ptr, priv_stack_alloc_sz);
>  		prog->aux->priv_stack_ptr = priv_stack_ptr;
> @@ -2079,10 +2069,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  	ctx.prog = prog;
>  
>  	ctx.offset = kvzalloc_objs(int, prog->len + 1);
> -	if (ctx.offset == NULL) {
> -		prog = orig_prog;
> +	if (ctx.offset == NULL)
>  		goto out_off;
> -	}
>  
>  	ctx.user_vm_start = bpf_arena_get_user_vm_start(prog->aux->arena);
>  	ctx.arena_vm_start = bpf_arena_get_kern_vm_start(prog->aux->arena);
> @@ -2095,15 +2083,11 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  	 * BPF line info needs ctx->offset[i] to be the offset of
>  	 * instruction[i] in jited image, so build prologue first.
>  	 */
> -	if (build_prologue(&ctx, was_classic)) {
> -		prog = orig_prog;
> +	if (build_prologue(&ctx, was_classic))
>  		goto out_off;
> -	}
>  
> -	if (build_body(&ctx, extra_pass)) {
> -		prog = orig_prog;
> +	if (build_body(&ctx, extra_pass))
>  		goto out_off;
> -	}
>  
>  	ctx.epilogue_offset = ctx.idx;
>  	build_epilogue(&ctx, was_classic);
> @@ -2121,10 +2105,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  	ro_header = bpf_jit_binary_pack_alloc(image_size, &ro_image_ptr,
>  					      sizeof(u64), &header, &image_ptr,
>  					      jit_fill_hole);
> -	if (!ro_header) {
> -		prog = orig_prog;
> +	if (!ro_header)
>  		goto out_off;
> -	}
>  
>  	/* Pass 2: Determine jited position and result for each instruction */
>  
> @@ -2152,10 +2134,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  	/* Dont write body instructions to memory for now */
>  	ctx.write = false;
>  
> -	if (build_body(&ctx, extra_pass)) {
> -		prog = orig_prog;
> +	if (build_body(&ctx, extra_pass))
>  		goto out_free_hdr;
> -	}
>  
>  	ctx.epilogue_offset = ctx.idx;
>  	ctx.exentry_idx = exentry_idx;
> @@ -2164,19 +2144,15 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  
>  	/* Pass 3: Adjust jump offset and write final image */
>  	if (build_body(&ctx, extra_pass) ||
> -		WARN_ON_ONCE(ctx.idx != ctx.epilogue_offset)) {
> -		prog = orig_prog;
> +		WARN_ON_ONCE(ctx.idx != ctx.epilogue_offset))
>  		goto out_free_hdr;
> -	}
>  
>  	build_epilogue(&ctx, was_classic);
>  	build_plt(&ctx);
>  
>  	/* Extra pass to validate JITed code. */
> -	if (validate_ctx(&ctx)) {
> -		prog = orig_prog;
> +	if (validate_ctx(&ctx))
>  		goto out_free_hdr;
> -	}
>  
>  	/* update the real prog size */
>  	prog_size = sizeof(u32) * ctx.idx;
> @@ -2193,15 +2169,13 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  		if (extra_pass && ctx.idx > jit_data->ctx.idx) {
>  			pr_err_once("multi-func JIT bug %d > %d\n",
>  				    ctx.idx, jit_data->ctx.idx);
> -			prog->bpf_func = NULL;
> -			prog->jited = 0;
> -			prog->jited_len = 0;
> +			clear_jit_state(prog);
>  			goto out_free_hdr;
>  		}
>  		if (WARN_ON(bpf_jit_binary_pack_finalize(ro_header, header))) {
>  			/* ro_header has been freed */
>  			ro_header = NULL;
> -			prog = orig_prog;
> +			clear_jit_state(prog);
>  			goto out_off;
>  		}
>  		/*
> @@ -2245,10 +2219,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  		kfree(jit_data);
>  		prog->aux->jit_data = NULL;
>  	}
> -out:
> -	if (tmp_blinded)
> -		bpf_jit_prog_release_other(prog, prog == orig_prog ?
> -					   tmp : orig_prog);
> +
>  	return prog;
>  
>  out_free_hdr:
> diff --git a/arch/loongarch/net/bpf_jit.c b/arch/loongarch/net/bpf_jit.c
> index 3bd89f55960d..57dd24d53c77 100644
> --- a/arch/loongarch/net/bpf_jit.c
> +++ b/arch/loongarch/net/bpf_jit.c
> @@ -1911,43 +1911,26 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
>  
>  struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  {
> -	bool tmp_blinded = false, extra_pass = false;
> +	bool extra_pass = false;
>  	u8 *image_ptr, *ro_image_ptr;
>  	int image_size, prog_size, extable_size;
>  	struct jit_ctx ctx;
>  	struct jit_data *jit_data;
>  	struct bpf_binary_header *header;
>  	struct bpf_binary_header *ro_header;
> -	struct bpf_prog *tmp, *orig_prog = prog;
>  
>  	/*
>  	 * If BPF JIT was not enabled then we must fall back to
>  	 * the interpreter.
>  	 */
>  	if (!prog->jit_requested)
> -		return orig_prog;
> -
> -	tmp = bpf_jit_blind_constants(prog);
> -	/*
> -	 * If blinding was requested and we failed during blinding,
> -	 * we must fall back to the interpreter. Otherwise, we save
> -	 * the new JITed code.
> -	 */
> -	if (IS_ERR(tmp))
> -		return orig_prog;
> -
> -	if (tmp != prog) {
> -		tmp_blinded = true;
> -		prog = tmp;
> -	}
> +		return prog;
>  
>  	jit_data = prog->aux->jit_data;
>  	if (!jit_data) {
>  		jit_data = kzalloc_obj(*jit_data);
> -		if (!jit_data) {
> -			prog = orig_prog;
> -			goto out;
> -		}
> +		if (!jit_data)
> +			return prog;
>  		prog->aux->jit_data = jit_data;
>  	}
>  	if (jit_data->ctx.offset) {
> @@ -1967,17 +1950,13 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  	ctx.user_vm_start = bpf_arena_get_user_vm_start(prog->aux->arena);
>  
>  	ctx.offset = kvcalloc(prog->len + 1, sizeof(u32), GFP_KERNEL);
> -	if (ctx.offset == NULL) {
> -		prog = orig_prog;
> +	if (ctx.offset == NULL)
>  		goto out_offset;
> -	}
>  
>  	/* 1. Initial fake pass to compute ctx->idx and set ctx->flags */
>  	build_prologue(&ctx);
> -	if (build_body(&ctx, extra_pass)) {
> -		prog = orig_prog;
> +	if (build_body(&ctx, extra_pass))
>  		goto out_offset;
> -	}
>  	ctx.epilogue_offset = ctx.idx;
>  	build_epilogue(&ctx);
>  
> @@ -1993,10 +1972,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  	/* Now we know the size of the structure to make */
>  	ro_header = bpf_jit_binary_pack_alloc(image_size, &ro_image_ptr, sizeof(u32),
>  					      &header, &image_ptr, jit_fill_hole);
> -	if (!ro_header) {
> -		prog = orig_prog;
> +	if (!ro_header)
>  		goto out_offset;
> -	}
>  
>  	/* 2. Now, the actual pass to generate final JIT code */
>  	/*
> @@ -2016,17 +1993,13 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  	ctx.num_exentries = 0;
>  
>  	build_prologue(&ctx);
> -	if (build_body(&ctx, extra_pass)) {
> -		prog = orig_prog;
> +	if (build_body(&ctx, extra_pass))
>  		goto out_free;
> -	}
>  	build_epilogue(&ctx);
>  
>  	/* 3. Extra pass to validate JITed code */
> -	if (validate_ctx(&ctx)) {
> -		prog = orig_prog;
> +	if (validate_ctx(&ctx))
>  		goto out_free;
> -	}
>  
>  	/* And we're done */
>  	if (bpf_jit_enable > 1)
> @@ -2041,7 +2014,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  		if (WARN_ON(bpf_jit_binary_pack_finalize(ro_header, header))) {
>  			/* ro_header has been freed */
>  			ro_header = NULL;
> -			prog = orig_prog;
>  			goto out_free;
>  		}
>  		/*
> @@ -2073,13 +2045,15 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  		prog->aux->jit_data = NULL;
>  	}
>  
> -out:
> -	if (tmp_blinded)
> -		bpf_jit_prog_release_other(prog, prog == orig_prog ? tmp : orig_prog);
> -
>  	return prog;
>  
>  out_free:
> +	if (prog->jited) {
> +		prog->bpf_func = NULL;
> +		prog->jited = 0;
> +		prog->jited_len = 0;
> +	}
> +
>  	if (header) {
>  		bpf_arch_text_copy(&ro_header->size, &header->size, sizeof(header->size));
>  		bpf_jit_binary_pack_free(ro_header, header);
> diff --git a/arch/mips/net/bpf_jit_comp.c b/arch/mips/net/bpf_jit_comp.c
> index e355dfca4400..d2b6c955f18e 100644
> --- a/arch/mips/net/bpf_jit_comp.c
> +++ b/arch/mips/net/bpf_jit_comp.c
> @@ -911,10 +911,8 @@ bool bpf_jit_needs_zext(void)
>  
>  struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  {
> -	struct bpf_prog *tmp, *orig_prog = prog;
>  	struct bpf_binary_header *header = NULL;
>  	struct jit_context ctx;
> -	bool tmp_blinded = false;
>  	unsigned int tmp_idx;
>  	unsigned int image_size;
>  	u8 *image_ptr;
> @@ -925,19 +923,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  	 * the interpreter.
>  	 */
>  	if (!prog->jit_requested)
> -		return orig_prog;
> -	/*
> -	 * If constant blinding was enabled and we failed during blinding
> -	 * then we must fall back to the interpreter. Otherwise, we save
> -	 * the new JITed code.
> -	 */
> -	tmp = bpf_jit_blind_constants(prog);
> -	if (IS_ERR(tmp))
> -		return orig_prog;
> -	if (tmp != prog) {
> -		tmp_blinded = true;
> -		prog = tmp;
> -	}
> +		return prog;
>  
>  	memset(&ctx, 0, sizeof(ctx));
>  	ctx.program = prog;
> @@ -1025,14 +1011,10 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  	prog->jited_len = image_size;
>  
>  out:
> -	if (tmp_blinded)
> -		bpf_jit_prog_release_other(prog, prog == orig_prog ?
> -					   tmp : orig_prog);
>  	kfree(ctx.descriptors);
>  	return prog;
>  
>  out_err:
> -	prog = orig_prog;
>  	if (header)
>  		bpf_jit_binary_free(header);
>  	goto out;
> diff --git a/arch/parisc/net/bpf_jit_core.c b/arch/parisc/net/bpf_jit_core.c
> index a5eb6b51e27a..4d339636a34a 100644
> --- a/arch/parisc/net/bpf_jit_core.c
> +++ b/arch/parisc/net/bpf_jit_core.c
> @@ -44,30 +44,19 @@ bool bpf_jit_needs_zext(void)
>  struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  {
>  	unsigned int prog_size = 0, extable_size = 0;
> -	bool tmp_blinded = false, extra_pass = false;
> -	struct bpf_prog *tmp, *orig_prog = prog;
> +	bool extra_pass = false;
>  	int pass = 0, prev_ninsns = 0, prologue_len, i;
>  	struct hppa_jit_data *jit_data;
>  	struct hppa_jit_context *ctx;
>  
>  	if (!prog->jit_requested)
> -		return orig_prog;
> -
> -	tmp = bpf_jit_blind_constants(prog);
> -	if (IS_ERR(tmp))
> -		return orig_prog;
> -	if (tmp != prog) {
> -		tmp_blinded = true;
> -		prog = tmp;
> -	}
> +		return prog;
>  
>  	jit_data = prog->aux->jit_data;
>  	if (!jit_data) {
>  		jit_data = kzalloc_obj(*jit_data);
> -		if (!jit_data) {
> -			prog = orig_prog;
> -			goto out;
> -		}
> +		if (!jit_data)
> +			return prog;
>  		prog->aux->jit_data = jit_data;
>  	}
>  
> @@ -81,10 +70,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  
>  	ctx->prog = prog;
>  	ctx->offset = kzalloc_objs(int, prog->len);
> -	if (!ctx->offset) {
> -		prog = orig_prog;
> +	if (!ctx->offset)
>  		goto out_offset;
> -	}
>  	for (i = 0; i < prog->len; i++) {
>  		prev_ninsns += 20;
>  		ctx->offset[i] = prev_ninsns;
> @@ -93,10 +80,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  	for (i = 0; i < NR_JIT_ITERATIONS; i++) {
>  		pass++;
>  		ctx->ninsns = 0;
> -		if (build_body(ctx, extra_pass, ctx->offset)) {
> -			prog = orig_prog;
> +		if (build_body(ctx, extra_pass, ctx->offset))
>  			goto out_offset;
> -		}
>  		ctx->body_len = ctx->ninsns;
>  		bpf_jit_build_prologue(ctx);
>  		ctx->prologue_len = ctx->ninsns - ctx->body_len;
> @@ -116,10 +101,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  						     &jit_data->image,
>  						     sizeof(long),
>  						     bpf_fill_ill_insns);
> -			if (!jit_data->header) {
> -				prog = orig_prog;
> +			if (!jit_data->header)
>  				goto out_offset;
> -			}
>  
>  			ctx->insns = (u32 *)jit_data->image;
>  			/*
> @@ -134,7 +117,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  		pr_err("bpf-jit: image did not converge in <%d passes!\n", i);
>  		if (jit_data->header)
>  			bpf_jit_binary_free(jit_data->header);
> -		prog = orig_prog;
>  		goto out_offset;
>  	}
>  
> @@ -148,7 +130,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  	bpf_jit_build_prologue(ctx);
>  	if (build_body(ctx, extra_pass, NULL)) {
>  		bpf_jit_binary_free(jit_data->header);
> -		prog = orig_prog;
>  		goto out_offset;
>  	}
>  	bpf_jit_build_epilogue(ctx);
> @@ -183,13 +164,10 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  		kfree(jit_data);
>  		prog->aux->jit_data = NULL;
>  	}
> -out:
> +
>  	if (HPPA_JIT_REBOOT)
>  		{ extern int machine_restart(char *); machine_restart(""); }
>  
> -	if (tmp_blinded)
> -		bpf_jit_prog_release_other(prog, prog == orig_prog ?
> -					   tmp : orig_prog);
>  	return prog;
>  }
>  
> diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_comp.c
> index 52162e4a7f84..7a7c49640a2f 100644
> --- a/arch/powerpc/net/bpf_jit_comp.c
> +++ b/arch/powerpc/net/bpf_jit_comp.c
> @@ -142,9 +142,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
>  	int flen;
>  	struct bpf_binary_header *fhdr = NULL;
>  	struct bpf_binary_header *hdr = NULL;
> -	struct bpf_prog *org_fp = fp;
> -	struct bpf_prog *tmp_fp;
> -	bool bpf_blinded = false;
>  	bool extra_pass = false;
>  	u8 *fimage = NULL;
>  	u32 *fcode_base;
> @@ -152,24 +149,13 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
>  	u32 fixup_len;
>  
>  	if (!fp->jit_requested)
> -		return org_fp;
> -
> -	tmp_fp = bpf_jit_blind_constants(org_fp);
> -	if (IS_ERR(tmp_fp))
> -		return org_fp;
> -
> -	if (tmp_fp != org_fp) {
> -		bpf_blinded = true;
> -		fp = tmp_fp;
> -	}
> +		return fp;
>  
>  	jit_data = fp->aux->jit_data;
>  	if (!jit_data) {
>  		jit_data = kzalloc_obj(*jit_data);
> -		if (!jit_data) {
> -			fp = org_fp;
> -			goto out;
> -		}
> +		if (!jit_data)
> +			return fp;
>  		fp->aux->jit_data = jit_data;
>  	}
>  
> @@ -194,10 +180,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
>  	}
>  
>  	addrs = kcalloc(flen + 1, sizeof(*addrs), GFP_KERNEL);
> -	if (addrs == NULL) {
> -		fp = org_fp;
> +	if (addrs == NULL)
>  		goto out_addrs;
> -	}
>  
>  	memset(&cgctx, 0, sizeof(struct codegen_context));
>  	bpf_jit_init_reg_mapping(&cgctx);
> @@ -211,11 +195,9 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
>  	cgctx.exception_cb = fp->aux->exception_cb;
>  
>  	/* Scouting faux-generate pass 0 */
> -	if (bpf_jit_build_body(fp, NULL, NULL, &cgctx, addrs, 0, false)) {
> +	if (bpf_jit_build_body(fp, NULL, NULL, &cgctx, addrs, 0, false))
>  		/* We hit something illegal or unsupported. */
> -		fp = org_fp;
>  		goto out_addrs;
> -	}
>  
>  	/*
>  	 * If we have seen a tail call, we need a second pass.
> @@ -226,10 +208,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
>  	 */
>  	if (cgctx.seen & SEEN_TAILCALL || !is_offset_in_branch_range((long)cgctx.idx * 4)) {
>  		cgctx.idx = 0;
> -		if (bpf_jit_build_body(fp, NULL, NULL, &cgctx, addrs, 0, false)) {
> -			fp = org_fp;
> +		if (bpf_jit_build_body(fp, NULL, NULL, &cgctx, addrs, 0, false))
>  			goto out_addrs;
> -		}
>  	}
>  
>  	bpf_jit_realloc_regs(&cgctx);
> @@ -250,10 +230,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
>  
>  	fhdr = bpf_jit_binary_pack_alloc(alloclen, &fimage, 4, &hdr, &image,
>  					      bpf_jit_fill_ill_insns);
> -	if (!fhdr) {
> -		fp = org_fp;
> +	if (!fhdr)
>  		goto out_addrs;
> -	}
>  
>  	if (extable_len)
>  		fp->aux->extable = (void *)fimage + FUNCTION_DESCR_SIZE + proglen + fixup_len;
> @@ -272,7 +250,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
>  				       extra_pass)) {
>  			bpf_arch_text_copy(&fhdr->size, &hdr->size, sizeof(hdr->size));
>  			bpf_jit_binary_pack_free(fhdr, hdr);
> -			fp = org_fp;
>  			goto out_addrs;
>  		}
>  		bpf_jit_build_epilogue(code_base, &cgctx);
> @@ -301,7 +278,9 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
>  
>  	if (!fp->is_func || extra_pass) {
>  		if (bpf_jit_binary_pack_finalize(fhdr, hdr)) {
> -			fp = org_fp;
> +			fp->bpf_func = NULL;
> +			fp->jited = 0;
> +			fp->jited_len = 0;
>  			goto out_addrs;
>  		}
>  		bpf_prog_fill_jited_linfo(fp, addrs);
> @@ -318,10 +297,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
>  		jit_data->hdr = hdr;
>  	}
>  
> -out:
> -	if (bpf_blinded)
> -		bpf_jit_prog_release_other(fp, fp == org_fp ? tmp_fp : org_fp);
> -
>  	return fp;
>  }
>  
> diff --git a/arch/riscv/net/bpf_jit_core.c b/arch/riscv/net/bpf_jit_core.c
> index b3581e926436..c77e8aba14d3 100644
> --- a/arch/riscv/net/bpf_jit_core.c
> +++ b/arch/riscv/net/bpf_jit_core.c
> @@ -44,29 +44,19 @@ bool bpf_jit_needs_zext(void)
>  struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  {
>  	unsigned int prog_size = 0, extable_size = 0;
> -	bool tmp_blinded = false, extra_pass = false;
> -	struct bpf_prog *tmp, *orig_prog = prog;
> +	bool extra_pass = false;
>  	int pass = 0, prev_ninsns = 0, i;
>  	struct rv_jit_data *jit_data;
>  	struct rv_jit_context *ctx;
>  
>  	if (!prog->jit_requested)
> -		return orig_prog;
> -
> -	tmp = bpf_jit_blind_constants(prog);
> -	if (IS_ERR(tmp))
> -		return orig_prog;
> -	if (tmp != prog) {
> -		tmp_blinded = true;
> -		prog = tmp;
> -	}
> +		return prog;
>  
>  	jit_data = prog->aux->jit_data;
>  	if (!jit_data) {
>  		jit_data = kzalloc_obj(*jit_data);
>  		if (!jit_data) {
> -			prog = orig_prog;
> -			goto out;
> +			return prog;
>  		}
>  		prog->aux->jit_data = jit_data;
>  	}
> @@ -83,15 +73,11 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  	ctx->user_vm_start = bpf_arena_get_user_vm_start(prog->aux->arena);
>  	ctx->prog = prog;
>  	ctx->offset = kzalloc_objs(int, prog->len);
> -	if (!ctx->offset) {
> -		prog = orig_prog;
> +	if (!ctx->offset)
>  		goto out_offset;
> -	}
>  
> -	if (build_body(ctx, extra_pass, NULL)) {
> -		prog = orig_prog;
> +	if (build_body(ctx, extra_pass, NULL))
>  		goto out_offset;
> -	}
>  
>  	for (i = 0; i < prog->len; i++) {
>  		prev_ninsns += 32;
> @@ -105,10 +91,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  		bpf_jit_build_prologue(ctx, bpf_is_subprog(prog));
>  		ctx->prologue_len = ctx->ninsns;
>  
> -		if (build_body(ctx, extra_pass, ctx->offset)) {
> -			prog = orig_prog;
> +		if (build_body(ctx, extra_pass, ctx->offset))
>  			goto out_offset;
> -		}
>  
>  		ctx->epilogue_offset = ctx->ninsns;
>  		bpf_jit_build_epilogue(ctx);
> @@ -126,10 +110,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  							  &jit_data->ro_image, sizeof(u32),
>  							  &jit_data->header, &jit_data->image,
>  							  bpf_fill_ill_insns);
> -			if (!jit_data->ro_header) {
> -				prog = orig_prog;
> +			if (!jit_data->ro_header)
>  				goto out_offset;
> -			}
>  
>  			/*
>  			 * Use the image(RW) for writing the JITed instructions. But also save
> @@ -150,7 +132,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  
>  	if (i == NR_JIT_ITERATIONS) {
>  		pr_err("bpf-jit: image did not converge in <%d passes!\n", i);
> -		prog = orig_prog;
>  		goto out_free_hdr;
>  	}
>  
> @@ -163,10 +144,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  	ctx->nexentries = 0;
>  
>  	bpf_jit_build_prologue(ctx, bpf_is_subprog(prog));
> -	if (build_body(ctx, extra_pass, NULL)) {
> -		prog = orig_prog;
> +	if (build_body(ctx, extra_pass, NULL))
>  		goto out_free_hdr;
> -	}
>  	bpf_jit_build_epilogue(ctx);
>  
>  	if (bpf_jit_enable > 1)
> @@ -180,7 +159,9 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  		if (WARN_ON(bpf_jit_binary_pack_finalize(jit_data->ro_header, jit_data->header))) {
>  			/* ro_header has been freed */
>  			jit_data->ro_header = NULL;
> -			prog = orig_prog;
> +			prog->bpf_func = NULL;
> +			prog->jited = 0;
> +			prog->jited_len = 0;
>  			goto out_offset;
>  		}
>  		/*
> @@ -198,11 +179,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  		kfree(jit_data);
>  		prog->aux->jit_data = NULL;
>  	}
> -out:
>  
> -	if (tmp_blinded)
> -		bpf_jit_prog_release_other(prog, prog == orig_prog ?
> -					   tmp : orig_prog);
>  	return prog;
>  
>  out_free_hdr:
> diff --git a/arch/s390/net/bpf_jit_comp.c b/arch/s390/net/bpf_jit_comp.c
> index 1f9a6b728beb..d6de2abfe4a7 100644
> --- a/arch/s390/net/bpf_jit_comp.c
> +++ b/arch/s390/net/bpf_jit_comp.c
> @@ -2305,36 +2305,20 @@ static struct bpf_binary_header *bpf_jit_alloc(struct bpf_jit *jit,
>   */
>  struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
>  {
> -	struct bpf_prog *tmp, *orig_fp = fp;
>  	struct bpf_binary_header *header;
>  	struct s390_jit_data *jit_data;
> -	bool tmp_blinded = false;
>  	bool extra_pass = false;
>  	struct bpf_jit jit;
>  	int pass;
>  
>  	if (!fp->jit_requested)
> -		return orig_fp;
> -
> -	tmp = bpf_jit_blind_constants(fp);
> -	/*
> -	 * If blinding was requested and we failed during blinding,
> -	 * we must fall back to the interpreter.
> -	 */
> -	if (IS_ERR(tmp))
> -		return orig_fp;
> -	if (tmp != fp) {
> -		tmp_blinded = true;
> -		fp = tmp;
> -	}
> +		return fp;
>  
>  	jit_data = fp->aux->jit_data;
>  	if (!jit_data) {
>  		jit_data = kzalloc_obj(*jit_data);
> -		if (!jit_data) {
> -			fp = orig_fp;
> -			goto out;
> -		}
> +		if (!jit_data)
> +			return fp;
>  		fp->aux->jit_data = jit_data;
>  	}
>  	if (jit_data->ctx.addrs) {
> @@ -2347,33 +2331,26 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
>  
>  	memset(&jit, 0, sizeof(jit));
>  	jit.addrs = kvcalloc(fp->len + 1, sizeof(*jit.addrs), GFP_KERNEL);
> -	if (jit.addrs == NULL) {
> -		fp = orig_fp;
> +	if (jit.addrs == NULL)
>  		goto free_addrs;
> -	}
>  	/*
>  	 * Three initial passes:
>  	 *   - 1/2: Determine clobbered registers
>  	 *   - 3:   Calculate program size and addrs array
>  	 */
>  	for (pass = 1; pass <= 3; pass++) {
> -		if (bpf_jit_prog(&jit, fp, extra_pass)) {
> -			fp = orig_fp;
> +		if (bpf_jit_prog(&jit, fp, extra_pass))
>  			goto free_addrs;
> -		}
>  	}
>  	/*
>  	 * Final pass: Allocate and generate program
>  	 */
>  	header = bpf_jit_alloc(&jit, fp);
> -	if (!header) {
> -		fp = orig_fp;
> +	if (!header)
>  		goto free_addrs;
> -	}
>  skip_init_ctx:
>  	if (bpf_jit_prog(&jit, fp, extra_pass)) {
>  		bpf_jit_binary_free(header);
> -		fp = orig_fp;
>  		goto free_addrs;
>  	}
>  	if (bpf_jit_enable > 1) {
> @@ -2383,7 +2360,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
>  	if (!fp->is_func || extra_pass) {
>  		if (bpf_jit_binary_lock_ro(header)) {
>  			bpf_jit_binary_free(header);
> -			fp = orig_fp;
>  			goto free_addrs;
>  		}
>  	} else {
> @@ -2402,10 +2378,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
>  		kfree(jit_data);
>  		fp->aux->jit_data = NULL;
>  	}
> -out:
> -	if (tmp_blinded)
> -		bpf_jit_prog_release_other(fp, fp == orig_fp ?
> -					   tmp : orig_fp);
> +
>  	return fp;
>  }
>  
> diff --git a/arch/sparc/net/bpf_jit_comp_64.c b/arch/sparc/net/bpf_jit_comp_64.c
> index b23d1c645ae5..86abd84d4005 100644
> --- a/arch/sparc/net/bpf_jit_comp_64.c
> +++ b/arch/sparc/net/bpf_jit_comp_64.c
> @@ -1479,37 +1479,22 @@ struct sparc64_jit_data {
>  
>  struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  {
> -	struct bpf_prog *tmp, *orig_prog = prog;
>  	struct sparc64_jit_data *jit_data;
>  	struct bpf_binary_header *header;
>  	u32 prev_image_size, image_size;
> -	bool tmp_blinded = false;
>  	bool extra_pass = false;
>  	struct jit_ctx ctx;
>  	u8 *image_ptr;
>  	int pass, i;
>  
>  	if (!prog->jit_requested)
> -		return orig_prog;
> -
> -	tmp = bpf_jit_blind_constants(prog);
> -	/* If blinding was requested and we failed during blinding,
> -	 * we must fall back to the interpreter.
> -	 */
> -	if (IS_ERR(tmp))
> -		return orig_prog;
> -	if (tmp != prog) {
> -		tmp_blinded = true;
> -		prog = tmp;
> -	}
> +		return prog;
>  
>  	jit_data = prog->aux->jit_data;
>  	if (!jit_data) {
>  		jit_data = kzalloc_obj(*jit_data);
> -		if (!jit_data) {
> -			prog = orig_prog;
> -			goto out;
> -		}
> +		if (!jit_data)
> +			return prog;
>  		prog->aux->jit_data = jit_data;
>  	}
>  	if (jit_data->ctx.offset) {
> @@ -1527,10 +1512,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  	ctx.prog = prog;
>  
>  	ctx.offset = kmalloc_array(prog->len, sizeof(unsigned int), GFP_KERNEL);
> -	if (ctx.offset == NULL) {
> -		prog = orig_prog;
> +	if (ctx.offset == NULL)
>  		goto out_off;
> -	}
>  
>  	/* Longest sequence emitted is for bswap32, 12 instructions.  Pre-cook
>  	 * the offset array so that we converge faster.
> @@ -1543,10 +1526,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  		ctx.idx = 0;
>  
>  		build_prologue(&ctx);
> -		if (build_body(&ctx)) {
> -			prog = orig_prog;
> +		if (build_body(&ctx))
>  			goto out_off;
> -		}
>  		build_epilogue(&ctx);
>  
>  		if (bpf_jit_enable > 1)
> @@ -1569,10 +1550,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  	image_size = sizeof(u32) * ctx.idx;
>  	header = bpf_jit_binary_alloc(image_size, &image_ptr,
>  				      sizeof(u32), jit_fill_hole);
> -	if (header == NULL) {
> -		prog = orig_prog;
> +	if (header == NULL)
>  		goto out_off;
> -	}
>  
>  	ctx.image = (u32 *)image_ptr;
>  skip_init_ctx:
> @@ -1582,7 +1561,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  
>  	if (build_body(&ctx)) {
>  		bpf_jit_binary_free(header);
> -		prog = orig_prog;
>  		goto out_off;
>  	}
>  
> @@ -1592,7 +1570,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  		pr_err("bpf_jit: Failed to converge, prev_size=%u size=%d\n",
>  		       prev_image_size, ctx.idx * 4);
>  		bpf_jit_binary_free(header);
> -		prog = orig_prog;
>  		goto out_off;
>  	}
>  
> @@ -1604,7 +1581,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  	if (!prog->is_func || extra_pass) {
>  		if (bpf_jit_binary_lock_ro(header)) {
>  			bpf_jit_binary_free(header);
> -			prog = orig_prog;
>  			goto out_off;
>  		}
>  	} else {
> @@ -1624,9 +1600,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  		kfree(jit_data);
>  		prog->aux->jit_data = NULL;
>  	}
> -out:
> -	if (tmp_blinded)
> -		bpf_jit_prog_release_other(prog, prog == orig_prog ?
> -					   tmp : orig_prog);
> +
>  	return prog;
>  }
> diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
> index e9b78040d703..de51ab3a11ee 100644
> --- a/arch/x86/net/bpf_jit_comp.c
> +++ b/arch/x86/net/bpf_jit_comp.c
> @@ -3717,13 +3717,11 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  {
>  	struct bpf_binary_header *rw_header = NULL;
>  	struct bpf_binary_header *header = NULL;
> -	struct bpf_prog *tmp, *orig_prog = prog;
>  	void __percpu *priv_stack_ptr = NULL;
>  	struct x64_jit_data *jit_data;
>  	int priv_stack_alloc_sz;
>  	int proglen, oldproglen = 0;
>  	struct jit_context ctx = {};
> -	bool tmp_blinded = false;
>  	bool extra_pass = false;
>  	bool padding = false;
>  	u8 *rw_image = NULL;
> @@ -3733,27 +3731,13 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  	int i;
>  
>  	if (!prog->jit_requested)
> -		return orig_prog;
> -
> -	tmp = bpf_jit_blind_constants(prog);
> -	/*
> -	 * If blinding was requested and we failed during blinding,
> -	 * we must fall back to the interpreter.
> -	 */
> -	if (IS_ERR(tmp))
> -		return orig_prog;
> -	if (tmp != prog) {
> -		tmp_blinded = true;
> -		prog = tmp;
> -	}
> +		return prog;
>  
>  	jit_data = prog->aux->jit_data;
>  	if (!jit_data) {
>  		jit_data = kzalloc_obj(*jit_data);
> -		if (!jit_data) {
> -			prog = orig_prog;
> +		if (!jit_data)
>  			goto out;
> -		}
>  		prog->aux->jit_data = jit_data;
>  	}
>  	priv_stack_ptr = prog->aux->priv_stack_ptr;
> @@ -3765,10 +3749,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  		priv_stack_alloc_sz = round_up(prog->aux->stack_depth, 8) +
>  				      2 * PRIV_STACK_GUARD_SZ;
>  		priv_stack_ptr = __alloc_percpu_gfp(priv_stack_alloc_sz, 8, GFP_KERNEL);
> -		if (!priv_stack_ptr) {
> -			prog = orig_prog;
> +		if (!priv_stack_ptr)
>  			goto out_priv_stack;
> -		}
>  
>  		priv_stack_init_guard(priv_stack_ptr, priv_stack_alloc_sz);
>  		prog->aux->priv_stack_ptr = priv_stack_ptr;
> @@ -3786,10 +3768,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  		goto skip_init_addrs;
>  	}
>  	addrs = kvmalloc_objs(*addrs, prog->len + 1);
> -	if (!addrs) {
> -		prog = orig_prog;
> +	if (!addrs)
>  		goto out_addrs;
> -	}
>  
>  	/*
>  	 * Before first pass, make a rough estimation of addrs[]
> @@ -3820,8 +3800,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  						   sizeof(rw_header->size));
>  				bpf_jit_binary_pack_free(header, rw_header);
>  			}
> -			/* Fall back to interpreter mode */
> -			prog = orig_prog;
>  			if (extra_pass) {
>  				prog->bpf_func = NULL;
>  				prog->jited = 0;
> @@ -3852,10 +3830,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  			header = bpf_jit_binary_pack_alloc(roundup(proglen, align) + extable_size,
>  							   &image, align, &rw_header, &rw_image,
>  							   jit_fill_hole);
> -			if (!header) {
> -				prog = orig_prog;
> +			if (!header)
>  				goto out_addrs;
> -			}
>  			prog->aux->extable = (void *) image + roundup(proglen, align);
>  		}
>  		oldproglen = proglen;
> @@ -3908,8 +3884,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  		prog->bpf_func = (void *)image + cfi_get_offset();
>  		prog->jited = 1;
>  		prog->jited_len = proglen - cfi_get_offset();
> -	} else {
> -		prog = orig_prog;
>  	}
>  
>  	if (!image || !prog->is_func || extra_pass) {
> @@ -3925,10 +3899,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  		kfree(jit_data);
>  		prog->aux->jit_data = NULL;
>  	}
> +
>  out:

small nit: is the label 'out' necessary now?

> -	if (tmp_blinded)
> -		bpf_jit_prog_release_other(prog, prog == orig_prog ?
> -					   tmp : orig_prog);
>  	return prog;
>  }
>  
> diff --git a/arch/x86/net/bpf_jit_comp32.c b/arch/x86/net/bpf_jit_comp32.c
> index dda423025c3d..5f259577614a 100644
> --- a/arch/x86/net/bpf_jit_comp32.c
> +++ b/arch/x86/net/bpf_jit_comp32.c
> @@ -2521,35 +2521,19 @@ bool bpf_jit_needs_zext(void)
>  struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  {
>  	struct bpf_binary_header *header = NULL;
> -	struct bpf_prog *tmp, *orig_prog = prog;
>  	int proglen, oldproglen = 0;
>  	struct jit_context ctx = {};
> -	bool tmp_blinded = false;
>  	u8 *image = NULL;
>  	int *addrs;
>  	int pass;
>  	int i;
>  
>  	if (!prog->jit_requested)
> -		return orig_prog;
> -
> -	tmp = bpf_jit_blind_constants(prog);
> -	/*
> -	 * If blinding was requested and we failed during blinding,
> -	 * we must fall back to the interpreter.
> -	 */
> -	if (IS_ERR(tmp))
> -		return orig_prog;
> -	if (tmp != prog) {
> -		tmp_blinded = true;
> -		prog = tmp;
> -	}
> +		return prog;
>  
>  	addrs = kmalloc_objs(*addrs, prog->len);
> -	if (!addrs) {
> -		prog = orig_prog;
> -		goto out;
> -	}
> +	if (!addrs)
> +		return prog;
>  
>  	/*
>  	 * Before first pass, make a rough estimation of addrs[]
> @@ -2574,7 +2558,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  			image = NULL;
>  			if (header)
>  				bpf_jit_binary_free(header);
> -			prog = orig_prog;
>  			goto out_addrs;
>  		}
>  		if (image) {
> @@ -2588,10 +2571,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  		if (proglen == oldproglen) {
>  			header = bpf_jit_binary_alloc(proglen, &image,
>  						      1, jit_fill_hole);
> -			if (!header) {
> -				prog = orig_prog;
> +			if (!header)
>  				goto out_addrs;
> -			}
>  		}
>  		oldproglen = proglen;
>  		cond_resched();
> @@ -2604,16 +2585,10 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>  		prog->bpf_func = (void *)image;
>  		prog->jited = 1;
>  		prog->jited_len = proglen;
> -	} else {
> -		prog = orig_prog;
>  	}
>  
>  out_addrs:
>  	kfree(addrs);
> -out:
> -	if (tmp_blinded)
> -		bpf_jit_prog_release_other(prog, prog == orig_prog ?
> -					   tmp : orig_prog);
>  	return prog;
>  }
>  
> diff --git a/include/linux/filter.h b/include/linux/filter.h
> index 44d7ae95ddbc..2484d85be63d 100644
> --- a/include/linux/filter.h
> +++ b/include/linux/filter.h
> @@ -1184,6 +1184,10 @@ static inline bool bpf_dump_raw_ok(const struct cred *cred)
>  
>  struct bpf_prog *bpf_patch_insn_single(struct bpf_prog *prog, u32 off,
>  				       const struct bpf_insn *patch, u32 len);
> +
> +struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, u32 off,
> +				     const struct bpf_insn *patch, u32 len);
> +
>  int bpf_remove_insns(struct bpf_prog *prog, u32 off, u32 cnt);
>  
>  static inline bool xdp_return_frame_no_direct(void)
> @@ -1310,8 +1314,7 @@ int bpf_jit_get_func_addr(const struct bpf_prog *prog,
>  
>  const char *bpf_jit_get_prog_name(struct bpf_prog *prog);
>  
> -struct bpf_prog *bpf_jit_blind_constants(struct bpf_prog *fp);
> -void bpf_jit_prog_release_other(struct bpf_prog *fp, struct bpf_prog *fp_other);
> +int bpf_jit_blind_constants(struct bpf_verifier_env *env);
>  
>  static inline void bpf_jit_dump(unsigned int flen, unsigned int proglen,
>  				u32 pass, void *image)
> @@ -1451,6 +1454,10 @@ static inline void bpf_prog_kallsyms_del(struct bpf_prog *fp)
>  {
>  }
>  
> +static inline int bpf_jit_blind_constants(struct bpf_verifier_env *env)
> +{
> +	return 0;
> +}
>  #endif /* CONFIG_BPF_JIT */
>  
>  void bpf_prog_kallsyms_del_all(struct bpf_prog *fp);
> diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
> index 229c74f3d6ae..c692213b1fdf 100644
> --- a/kernel/bpf/core.c
> +++ b/kernel/bpf/core.c
> @@ -1427,82 +1427,19 @@ static int bpf_jit_blind_insn(const struct bpf_insn *from,
>  	return to - to_buff;
>  }
>  
> -static struct bpf_prog *bpf_prog_clone_create(struct bpf_prog *fp_other,
> -					      gfp_t gfp_extra_flags)
> -{
> -	gfp_t gfp_flags = GFP_KERNEL | __GFP_ZERO | gfp_extra_flags;
> -	struct bpf_prog *fp;
> -
> -	fp = __vmalloc(fp_other->pages * PAGE_SIZE, gfp_flags);
> -	if (fp != NULL) {
> -		/* aux->prog still points to the fp_other one, so
> -		 * when promoting the clone to the real program,
> -		 * this still needs to be adapted.
> -		 */
> -		memcpy(fp, fp_other, fp_other->pages * PAGE_SIZE);
> -	}
> -
> -	return fp;
> -}
> -
> -static void bpf_prog_clone_free(struct bpf_prog *fp)
> -{
> -	/* aux was stolen by the other clone, so we cannot free
> -	 * it from this path! It will be freed eventually by the
> -	 * other program on release.
> -	 *
> -	 * At this point, we don't need a deferred release since
> -	 * clone is guaranteed to not be locked.
> -	 */
> -	fp->aux = NULL;
> -	fp->stats = NULL;
> -	fp->active = NULL;
> -	__bpf_prog_free(fp);
> -}
> -
> -void bpf_jit_prog_release_other(struct bpf_prog *fp, struct bpf_prog *fp_other)
> -{
> -	/* We have to repoint aux->prog to self, as we don't
> -	 * know whether fp here is the clone or the original.
> -	 */
> -	fp->aux->prog = fp;
> -	bpf_prog_clone_free(fp_other);
> -}
> -
> -static void adjust_insn_arrays(struct bpf_prog *prog, u32 off, u32 len)
> -{
> -#ifdef CONFIG_BPF_SYSCALL
> -	struct bpf_map *map;
> -	int i;
> -
> -	if (len <= 1)
> -		return;
> -
> -	for (i = 0; i < prog->aux->used_map_cnt; i++) {
> -		map = prog->aux->used_maps[i];
> -		if (map->map_type == BPF_MAP_TYPE_INSN_ARRAY)
> -			bpf_insn_array_adjust(map, off, len);
> -	}
> -#endif
> -}
> -
> -struct bpf_prog *bpf_jit_blind_constants(struct bpf_prog *prog)
> +int bpf_jit_blind_constants(struct bpf_verifier_env *env)
>  {
>  	struct bpf_insn insn_buff[16], aux[2];
> -	struct bpf_prog *clone, *tmp;
> +	struct bpf_prog *prog = env->prog;
>  	int insn_delta, insn_cnt;
>  	struct bpf_insn *insn;
>  	int i, rewritten;
>  
>  	if (!prog->blinding_requested || prog->blinded)
> -		return prog;
> -
> -	clone = bpf_prog_clone_create(prog, GFP_USER);
> -	if (!clone)
> -		return ERR_PTR(-ENOMEM);
> +		return 0;
>  
> -	insn_cnt = clone->len;
> -	insn = clone->insnsi;
> +	insn_cnt = prog->len;
> +	insn = prog->insnsi;
>  
>  	for (i = 0; i < insn_cnt; i++, insn++) {
>  		if (bpf_pseudo_func(insn)) {
> @@ -1523,35 +1460,25 @@ struct bpf_prog *bpf_jit_blind_constants(struct bpf_prog *prog)
>  		    insn[1].code == 0)
>  			memcpy(aux, insn, sizeof(aux));
>  
> -		rewritten = bpf_jit_blind_insn(insn, aux, insn_buff,
> -						clone->aux->verifier_zext);
> +		rewritten = bpf_jit_blind_insn(insn, aux, insn_buff, prog->aux->verifier_zext);
>  		if (!rewritten)
>  			continue;
>  
> -		tmp = bpf_patch_insn_single(clone, i, insn_buff, rewritten);
> -		if (IS_ERR(tmp)) {
> -			/* Patching may have repointed aux->prog during
> -			 * realloc from the original one, so we need to
> -			 * fix it up here on error.
> -			 */
> -			bpf_jit_prog_release_other(prog, clone);
> -			return tmp;
> -		}
> +		prog = bpf_patch_insn_data(env, i, insn_buff, rewritten);
> +		if (!prog)
> +			return -ENOMEM;
>  
> -		clone = tmp;
> +		env->prog = prog;
>  		insn_delta = rewritten - 1;
>  
> -		/* Instructions arrays must be updated using absolute xlated offsets */
> -		adjust_insn_arrays(clone, prog->aux->subprog_start + i, rewritten);
> -
>  		/* Walk new program and skip insns we just inserted. */
> -		insn = clone->insnsi + i + insn_delta;
> +		insn = prog->insnsi + i + insn_delta;
>  		insn_cnt += insn_delta;
>  		i        += insn_delta;
>  	}
>  
> -	clone->blinded = 1;
> -	return clone;
> +	prog->blinded = 1;
> +	return 0;
>  }
>  #endif /* CONFIG_BPF_JIT */
>  
> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index 7aa06f534cb2..e290c9b7d13d 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
> @@ -22070,8 +22070,8 @@ static void adjust_poke_descs(struct bpf_prog *prog, u32 off, u32 len)
>  	}
>  }
>  
> -static struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, u32 off,
> -					    const struct bpf_insn *patch, u32 len)
> +struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, u32 off,
> +				     const struct bpf_insn *patch, u32 len)
>  {
>  	struct bpf_prog *new_prog;
>  	struct bpf_insn_aux_data *new_data = NULL;
> @@ -22846,7 +22846,6 @@ static int jit_subprogs(struct bpf_verifier_env *env)
>  	struct bpf_insn *insn;
>  	void *old_bpf_func;
>  	int err, num_exentries;
> -	int old_len, subprog_start_adjustment = 0;

nice :)

>  
>  	if (env->subprog_cnt <= 1)
>  		return 0;
> @@ -22918,10 +22917,11 @@ static int jit_subprogs(struct bpf_verifier_env *env)
>  			goto out_free;
>  		func[i]->is_func = 1;
>  		func[i]->sleepable = prog->sleepable;
> +		func[i]->blinded = prog->blinded;
>  		func[i]->aux->func_idx = i;
>  		/* Below members will be freed only at prog->aux */
>  		func[i]->aux->btf = prog->aux->btf;
> -		func[i]->aux->subprog_start = subprog_start + subprog_start_adjustment;
> +		func[i]->aux->subprog_start = subprog_start;
>  		func[i]->aux->func_info = prog->aux->func_info;
>  		func[i]->aux->func_info_cnt = prog->aux->func_info_cnt;
>  		func[i]->aux->poke_tab = prog->aux->poke_tab;
> @@ -22977,15 +22977,7 @@ static int jit_subprogs(struct bpf_verifier_env *env)
>  		func[i]->aux->might_sleep = env->subprog_info[i].might_sleep;
>  		if (!i)
>  			func[i]->aux->exception_boundary = env->seen_exception;
> -
> -		/*
> -		 * To properly pass the absolute subprog start to jit
> -		 * all instruction adjustments should be accumulated
> -		 */
> -		old_len = func[i]->len;
>  		func[i] = bpf_int_jit_compile(func[i]);
> -		subprog_start_adjustment += func[i]->len - old_len;
> -
>  		if (!func[i]->jited) {
>  			err = -ENOTSUPP;
>  			goto out_free;
> @@ -23136,6 +23128,9 @@ static int fixup_call_args(struct bpf_verifier_env *env)
>  
>  	if (env->prog->jit_requested &&
>  	    !bpf_prog_is_offloaded(env->prog->aux)) {
> +		err = bpf_jit_blind_constants(env);
> +		if (err)
> +			return err;
>  		err = jit_subprogs(env);
>  		if (err == 0)
>  			return 0;
> -- 
> 2.47.3
> 

Reviewed-by: Anton Protopopov <a.s.protopopov@gmail.com>


^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [bpf-next v8 1/5] bpf: Move constants blinding from JIT to verifier
  2026-03-09 17:20   ` Anton Protopopov
@ 2026-03-10  6:52     ` Xu Kuohai
  0 siblings, 0 replies; 20+ messages in thread
From: Xu Kuohai @ 2026-03-10  6:52 UTC (permalink / raw)
  To: Anton Protopopov
  Cc: bpf, linux-kernel, linux-arm-kernel, Alexei Starovoitov,
	Daniel Borkmann, Andrii Nakryiko, Martin KaFai Lau,
	Eduard Zingerman, Yonghong Song, Puranjay Mohan, Shahab Vahedi,
	Russell King, Tiezhu Yang, Hengqi Chen, Johan Almbladh,
	Paul Burton, Hari Bathini, Christophe Leroy, Naveen N Rao,
	Luke Nelson, Xi Wang, Björn Töpel, Pu Lehui,
	Ilya Leoshkevich, Heiko Carstens, Vasily Gorbik, David S . Miller,
	Wang YanQing

On 3/10/2026 1:20 AM, Anton Protopopov wrote:

[...]

>>   	if (!jit_data) {
>>   		jit_data = kzalloc_obj(*jit_data);
>> -		if (!jit_data) {
>> -			prog = orig_prog;
>> +		if (!jit_data)
>>   			goto out;
>> -		}
>>   		prog->aux->jit_data = jit_data;
>>   	}
>>   	priv_stack_ptr = prog->aux->priv_stack_ptr;
>> @@ -3765,10 +3749,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>>   		priv_stack_alloc_sz = round_up(prog->aux->stack_depth, 8) +
>>   				      2 * PRIV_STACK_GUARD_SZ;
>>   		priv_stack_ptr = __alloc_percpu_gfp(priv_stack_alloc_sz, 8, GFP_KERNEL);
>> -		if (!priv_stack_ptr) {
>> -			prog = orig_prog;
>> +		if (!priv_stack_ptr)
>>   			goto out_priv_stack;
>> -		}
>>   
>>   		priv_stack_init_guard(priv_stack_ptr, priv_stack_alloc_sz);
>>   		prog->aux->priv_stack_ptr = priv_stack_ptr;
>> @@ -3786,10 +3768,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>>   		goto skip_init_addrs;
>>   	}
>>   	addrs = kvmalloc_objs(*addrs, prog->len + 1);
>> -	if (!addrs) {
>> -		prog = orig_prog;
>> +	if (!addrs)
>>   		goto out_addrs;
>> -	}
>>   
>>   	/*
>>   	 * Before first pass, make a rough estimation of addrs[]
>> @@ -3820,8 +3800,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>>   						   sizeof(rw_header->size));
>>   				bpf_jit_binary_pack_free(header, rw_header);
>>   			}
>> -			/* Fall back to interpreter mode */
>> -			prog = orig_prog;
>>   			if (extra_pass) {
>>   				prog->bpf_func = NULL;
>>   				prog->jited = 0;
>> @@ -3852,10 +3830,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>>   			header = bpf_jit_binary_pack_alloc(roundup(proglen, align) + extable_size,
>>   							   &image, align, &rw_header, &rw_image,
>>   							   jit_fill_hole);
>> -			if (!header) {
>> -				prog = orig_prog;
>> +			if (!header)
>>   				goto out_addrs;
>> -			}
>>   			prog->aux->extable = (void *) image + roundup(proglen, align);
>>   		}
>>   		oldproglen = proglen;
>> @@ -3908,8 +3884,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>>   		prog->bpf_func = (void *)image + cfi_get_offset();
>>   		prog->jited = 1;
>>   		prog->jited_len = proglen - cfi_get_offset();
>> -	} else {
>> -		prog = orig_prog;
>>   	}
>>   
>>   	if (!image || !prog->is_func || extra_pass) {
>> @@ -3925,10 +3899,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>>   		kfree(jit_data);
>>   		prog->aux->jit_data = NULL;
>>   	}
>> +
>>   out:
> 
> small nit: is the label 'out' necessary now?
> 

There is a "goto out" in the "if (!jit_data)" body, but it is the
only one, I'll replace it with a direct "return prog" and remove
the out label.



^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [bpf-next v8 1/5] bpf: Move constants blinding from JIT to verifier
  2026-03-09 14:00 ` [bpf-next v8 1/5] bpf: Move constants blinding from JIT to verifier Xu Kuohai
  2026-03-09 17:20   ` Anton Protopopov
@ 2026-03-09 21:25   ` Eduard Zingerman
  2026-03-10  7:39     ` Xu Kuohai
  2026-03-17 10:55   ` kernel test robot
  2 siblings, 1 reply; 20+ messages in thread
From: Eduard Zingerman @ 2026-03-09 21:25 UTC (permalink / raw)
  To: Xu Kuohai, bpf, linux-kernel, linux-arm-kernel
  Cc: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
	Martin KaFai Lau, Yonghong Song, Puranjay Mohan, Anton Protopopov,
	Shahab Vahedi, Russell King, Tiezhu Yang, Hengqi Chen,
	Johan Almbladh, Paul Burton, Hari Bathini, Christophe Leroy,
	Naveen N Rao, Luke Nelson, Xi Wang, Björn Töpel,
	Pu Lehui, Ilya Leoshkevich, Heiko Carstens, Vasily Gorbik,
	David S . Miller, Wang YanQing

On Mon, 2026-03-09 at 22:00 +0800, Xu Kuohai wrote:

This was discussed some time ago in [1]. In that message Daniel notes
the following:

  > constant blinding needs to work from native bpf(2) as well as from
  > cbpf->ebpf (seccomp-bpf, filters, etc)

It appears this patch does not address the cbpf->ebpf part, or did I
miss something?

[1] https://lore.kernel.org/bpf/284404c7-c6e0-4cf9-8ada-71ebfc681541@iogearbox.net/

[...]

> diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_comp.c
> index 52162e4a7f84..7a7c49640a2f 100644
> --- a/arch/powerpc/net/bpf_jit_comp.c
> +++ b/arch/powerpc/net/bpf_jit_comp.c

[...]

> @@ -272,7 +250,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
>  				       extra_pass)) {
>  			bpf_arch_text_copy(&fhdr->size, &hdr->size, sizeof(hdr->size));
>  			bpf_jit_binary_pack_free(fhdr, hdr);
> -			fp = org_fp;

Is it necessary to set `...; fp->jited = 0; ...` here?
It seems it does if extra_pass is set.

>  			goto out_addrs;
>  		}
>  		bpf_jit_build_epilogue(code_base, &cgctx);
> @@ -301,7 +278,9 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
>  
>  	if (!fp->is_func || extra_pass) {
>  		if (bpf_jit_binary_pack_finalize(fhdr, hdr)) {
> -			fp = org_fp;
> +			fp->bpf_func = NULL;
> +			fp->jited = 0;
> +			fp->jited_len = 0;
>  			goto out_addrs;
>  		}
>  		bpf_prog_fill_jited_linfo(fp, addrs);

[...]

> diff --git a/arch/s390/net/bpf_jit_comp.c b/arch/s390/net/bpf_jit_comp.c
> index 1f9a6b728beb..d6de2abfe4a7 100644
> --- a/arch/s390/net/bpf_jit_comp.c
> +++ b/arch/s390/net/bpf_jit_comp.c

[...]

> @@ -2383,7 +2360,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
>  	if (!fp->is_func || extra_pass) {
>  		if (bpf_jit_binary_lock_ro(header)) {
>  			bpf_jit_binary_free(header);
> -			fp = orig_fp;

In the similar condition for powerpc you reset `...; prog->jited = 0; ...`,
is this case different?

>  			goto free_addrs;
>  		}
>  	} else {

[...]


^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [bpf-next v8 1/5] bpf: Move constants blinding from JIT to verifier
  2026-03-09 21:25   ` Eduard Zingerman
@ 2026-03-10  7:39     ` Xu Kuohai
  0 siblings, 0 replies; 20+ messages in thread
From: Xu Kuohai @ 2026-03-10  7:39 UTC (permalink / raw)
  To: Eduard Zingerman, bpf, linux-kernel, linux-arm-kernel
  Cc: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
	Martin KaFai Lau, Yonghong Song, Puranjay Mohan, Anton Protopopov,
	Shahab Vahedi, Russell King, Tiezhu Yang, Hengqi Chen,
	Johan Almbladh, Paul Burton, Hari Bathini, Christophe Leroy,
	Naveen N Rao, Luke Nelson, Xi Wang, Björn Töpel,
	Pu Lehui, Ilya Leoshkevich, Heiko Carstens, Vasily Gorbik,
	David S . Miller, Wang YanQing

On 3/10/2026 5:25 AM, Eduard Zingerman wrote:
> On Mon, 2026-03-09 at 22:00 +0800, Xu Kuohai wrote:
> 
> This was discussed some time ago in [1]. In that message Daniel notes
> the following:
> 
>    > constant blinding needs to work from native bpf(2) as well as from
>    > cbpf->ebpf (seccomp-bpf, filters, etc)
> 
> It appears this patch does not address the cbpf->ebpf part, or did I
> miss something?
> 
> [1] https://lore.kernel.org/bpf/284404c7-c6e0-4cf9-8ada-71ebfc681541@iogearbox.net/
>

Hmm, I overlooked that the cbpf jit also needs constant blinding. I'll add it back
in bpf_prog_select_runtime() before the JIT is invoked.

> [...]
> 
>> diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_comp.c
>> index 52162e4a7f84..7a7c49640a2f 100644
>> --- a/arch/powerpc/net/bpf_jit_comp.c
>> +++ b/arch/powerpc/net/bpf_jit_comp.c
> 
> [...]
> 
>> @@ -272,7 +250,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
>>   				       extra_pass)) {
>>   			bpf_arch_text_copy(&fhdr->size, &hdr->size, sizeof(hdr->size));
>>   			bpf_jit_binary_pack_free(fhdr, hdr);
>> -			fp = org_fp;
> 
> Is it necessary to set `...; fp->jited = 0; ...` here?
> It seems it does if extra_pass is set.
> 
Yes. When extra_pass is set, orig_fp and fp are the same, so the original
fp = orig_fp cannot restore fp either.

>>   			goto out_addrs;
>>   		}
>>   		bpf_jit_build_epilogue(code_base, &cgctx);
>> @@ -301,7 +278,9 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
>>   
>>   	if (!fp->is_func || extra_pass) {
>>   		if (bpf_jit_binary_pack_finalize(fhdr, hdr)) {
>> -			fp = org_fp;
>> +			fp->bpf_func = NULL;
>> +			fp->jited = 0;
>> +			fp->jited_len = 0;
>>   			goto out_addrs;
>>   		}
>>   		bpf_prog_fill_jited_linfo(fp, addrs);
> 
> [...]
> 
>> diff --git a/arch/s390/net/bpf_jit_comp.c b/arch/s390/net/bpf_jit_comp.c
>> index 1f9a6b728beb..d6de2abfe4a7 100644
>> --- a/arch/s390/net/bpf_jit_comp.c
>> +++ b/arch/s390/net/bpf_jit_comp.c
> 
> [...]
> 
>> @@ -2383,7 +2360,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
>>   	if (!fp->is_func || extra_pass) {
>>   		if (bpf_jit_binary_lock_ro(header)) {
>>   			bpf_jit_binary_free(header);
>> -			fp = orig_fp;
> 
> In the similar condition for powerpc you reset `...; prog->jited = 0; ...`,
> is this case different?
>

No difference, will fix it.

>>   			goto free_addrs;
>>   		}
>>   	} else {
> 
> [...]



^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [bpf-next v8 1/5] bpf: Move constants blinding from JIT to verifier
  2026-03-09 14:00 ` [bpf-next v8 1/5] bpf: Move constants blinding from JIT to verifier Xu Kuohai
  2026-03-09 17:20   ` Anton Protopopov
  2026-03-09 21:25   ` Eduard Zingerman
@ 2026-03-17 10:55   ` kernel test robot
  2 siblings, 0 replies; 20+ messages in thread
From: kernel test robot @ 2026-03-17 10:55 UTC (permalink / raw)
  To: Xu Kuohai, bpf, linux-kernel, linux-arm-kernel
  Cc: oe-kbuild-all, Alexei Starovoitov, Daniel Borkmann,
	Andrii Nakryiko, Martin KaFai Lau, Eduard Zingerman,
	Yonghong Song, Puranjay Mohan, Anton Protopopov, Shahab Vahedi,
	Russell King, Tiezhu Yang, Hengqi Chen, Johan Almbladh,
	Paul Burton, Hari Bathini, Christophe Leroy, Naveen N Rao,
	Luke Nelson, Xi Wang, Björn Töpel, Pu Lehui,
	Ilya Leoshkevich, Heiko Carstens, Vasily Gorbik, David S . Miller,
	Wang YanQing

Hi Xu,

kernel test robot noticed the following build errors:

[auto build test ERROR on bpf-next/master]

url:    https://github.com/intel-lab-lkp/linux/commits/Xu-Kuohai/bpf-Move-constants-blinding-from-JIT-to-verifier/20260309-214106
base:   https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git master
patch link:    https://lore.kernel.org/r/20260309140044.2652538-2-xukuohai%40huaweicloud.com
patch subject: [bpf-next v8 1/5] bpf: Move constants blinding from JIT to verifier
config: powerpc64-randconfig-001-20260317 (https://download.01.org/0day-ci/archive/20260317/202603171843.yv0X1mn3-lkp@intel.com/config)
compiler: powerpc64-linux-gcc (GCC) 8.5.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260317/202603171843.yv0X1mn3-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202603171843.yv0X1mn3-lkp@intel.com/

All errors (new ones prefixed by >>):

   powerpc64-linux-ld: kernel/bpf/core.o: in function `bpf_jit_blind_constants':
>> kernel/bpf/core.c:1467: undefined reference to `bpf_patch_insn_data'


vim +1467 kernel/bpf/core.c

  1429	
  1430	int bpf_jit_blind_constants(struct bpf_verifier_env *env)
  1431	{
  1432		struct bpf_insn insn_buff[16], aux[2];
  1433		struct bpf_prog *prog = env->prog;
  1434		int insn_delta, insn_cnt;
  1435		struct bpf_insn *insn;
  1436		int i, rewritten;
  1437	
  1438		if (!prog->blinding_requested || prog->blinded)
  1439			return 0;
  1440	
  1441		insn_cnt = prog->len;
  1442		insn = prog->insnsi;
  1443	
  1444		for (i = 0; i < insn_cnt; i++, insn++) {
  1445			if (bpf_pseudo_func(insn)) {
  1446				/* ld_imm64 with an address of bpf subprog is not
  1447				 * a user controlled constant. Don't randomize it,
  1448				 * since it will conflict with jit_subprogs() logic.
  1449				 */
  1450				insn++;
  1451				i++;
  1452				continue;
  1453			}
  1454	
  1455			/* We temporarily need to hold the original ld64 insn
  1456			 * so that we can still access the first part in the
  1457			 * second blinding run.
  1458			 */
  1459			if (insn[0].code == (BPF_LD | BPF_IMM | BPF_DW) &&
  1460			    insn[1].code == 0)
  1461				memcpy(aux, insn, sizeof(aux));
  1462	
  1463			rewritten = bpf_jit_blind_insn(insn, aux, insn_buff, prog->aux->verifier_zext);
  1464			if (!rewritten)
  1465				continue;
  1466	
> 1467			prog = bpf_patch_insn_data(env, i, insn_buff, rewritten);
  1468			if (!prog)
  1469				return -ENOMEM;
  1470	
  1471			env->prog = prog;
  1472			insn_delta = rewritten - 1;
  1473	
  1474			/* Walk new program and skip insns we just inserted. */
  1475			insn = prog->insnsi + i + insn_delta;
  1476			insn_cnt += insn_delta;
  1477			i        += insn_delta;
  1478		}
  1479	
  1480		prog->blinded = 1;
  1481		return 0;
  1482	}
  1483	#endif /* CONFIG_BPF_JIT */
  1484	

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki


^ permalink raw reply	[flat|nested] 20+ messages in thread

* [bpf-next v8 2/5] bpf: Pass bpf_verifier_env to JIT
  2026-03-09 14:00 [bpf-next v8 0/5] emit ENDBR/BTI instructions for indirect jump targets Xu Kuohai
  2026-03-09 14:00 ` [bpf-next v8 1/5] bpf: Move constants blinding from JIT to verifier Xu Kuohai
@ 2026-03-09 14:00 ` Xu Kuohai
  2026-03-09 16:56   ` Anton Protopopov
  2026-03-09 14:00 ` [bpf-next v8 3/5] bpf: Add helper to detect indirect jump targets Xu Kuohai
                   ` (4 subsequent siblings)
  6 siblings, 1 reply; 20+ messages in thread
From: Xu Kuohai @ 2026-03-09 14:00 UTC (permalink / raw)
  To: bpf, linux-kernel, linux-arm-kernel
  Cc: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
	Martin KaFai Lau, Eduard Zingerman, Yonghong Song, Puranjay Mohan,
	Anton Protopopov, Shahab Vahedi, Russell King, Tiezhu Yang,
	Hengqi Chen, Johan Almbladh, Paul Burton, Hari Bathini,
	Christophe Leroy, Naveen N Rao, Luke Nelson, Xi Wang,
	Björn Töpel, Pu Lehui, Ilya Leoshkevich, Heiko Carstens,
	Vasily Gorbik, David S . Miller, Wang YanQing

From: Xu Kuohai <xukuohai@huawei.com>

Pass bpf_verifier_env to bpf_int_jit_compile(). The follow-up patch will
use env->insn_aux_data in the JIT stage to detect indirect jump targets.

Since bpf_prog_select_runtime() can be called by cbpf and test code
without verifier, introduce helper function __bpf_prog_select_runtime()
to accept the env parameter.

Remove the call to bpf_prog_select_runtime() in bpf_prog_load(), and
switch to call __bpf_prog_select_runtime() in the verifier, with env
variable passed. The original bpf_prog_select_runtime() is preserved for
cbpf and test code, where env is NULL.

Signed-off-by: Xu Kuohai <xukuohai@huawei.com>
---
 arch/arc/net/bpf_jit_core.c      | 17 ++++++++--------
 arch/arm/net/bpf_jit_32.c        |  2 +-
 arch/arm64/net/bpf_jit_comp.c    |  2 +-
 arch/loongarch/net/bpf_jit.c     |  2 +-
 arch/mips/net/bpf_jit_comp.c     |  2 +-
 arch/parisc/net/bpf_jit_core.c   |  2 +-
 arch/powerpc/net/bpf_jit_comp.c  |  2 +-
 arch/riscv/net/bpf_jit_core.c    |  2 +-
 arch/s390/net/bpf_jit_comp.c     |  2 +-
 arch/sparc/net/bpf_jit_comp_64.c |  2 +-
 arch/x86/net/bpf_jit_comp.c      |  2 +-
 arch/x86/net/bpf_jit_comp32.c    |  2 +-
 include/linux/filter.h           |  4 +++-
 kernel/bpf/core.c                | 34 +++++++++++++++++++-------------
 kernel/bpf/syscall.c             |  4 ----
 kernel/bpf/verifier.c            |  8 ++++++--
 16 files changed, 49 insertions(+), 40 deletions(-)

diff --git a/arch/arc/net/bpf_jit_core.c b/arch/arc/net/bpf_jit_core.c
index 12facf5750da..cf708c308adb 100644
--- a/arch/arc/net/bpf_jit_core.c
+++ b/arch/arc/net/bpf_jit_core.c
@@ -153,7 +153,8 @@ static void jit_dump(const struct jit_context *ctx)
 }
 
 /* Initialise the context so there's no garbage. */
-static int jit_ctx_init(struct jit_context *ctx, struct bpf_prog *prog)
+static int jit_ctx_init(struct jit_context *ctx, struct bpf_verifier_env *env,
+			struct bpf_prog *prog)
 {
 	memset(ctx, 0, sizeof(*ctx));
 
@@ -1317,7 +1318,7 @@ static int jit_patch_relocations(struct jit_context *ctx)
  * to get the necessary data for the real compilation phase,
  * jit_compile().
  */
-static struct bpf_prog *do_normal_pass(struct bpf_prog *prog)
+static struct bpf_prog *do_normal_pass(struct bpf_verifier_env *env, struct bpf_prog *prog)
 {
 	struct jit_context ctx;
 
@@ -1325,7 +1326,7 @@ static struct bpf_prog *do_normal_pass(struct bpf_prog *prog)
 	if (!prog->jit_requested)
 		return prog;
 
-	if (jit_ctx_init(&ctx, prog)) {
+	if (jit_ctx_init(&ctx, env, prog)) {
 		jit_ctx_cleanup(&ctx);
 		return prog;
 	}
@@ -1356,7 +1357,7 @@ static struct bpf_prog *do_normal_pass(struct bpf_prog *prog)
  * again to get the newly translated addresses in order to resolve
  * the "call"s.
  */
-static struct bpf_prog *do_extra_pass(struct bpf_prog *prog)
+static struct bpf_prog *do_extra_pass(struct bpf_verifier_env *env, struct bpf_prog *prog)
 {
 	struct jit_context ctx;
 
@@ -1364,7 +1365,7 @@ static struct bpf_prog *do_extra_pass(struct bpf_prog *prog)
 	if (check_jit_context(prog))
 		return prog;
 
-	if (jit_ctx_init(&ctx, prog)) {
+	if (jit_ctx_init(&ctx, env, prog)) {
 		jit_ctx_cleanup(&ctx);
 		return prog;
 	}
@@ -1393,15 +1394,15 @@ static struct bpf_prog *do_extra_pass(struct bpf_prog *prog)
  * (re)locations involved that their addresses are not known
  * during the first run.
  */
-struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
+struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog)
 {
 	vm_dump(prog);
 
 	/* Was this program already translated? */
 	if (!prog->jited)
-		return do_normal_pass(prog);
+		return do_normal_pass(env, prog);
 	else
-		return do_extra_pass(prog);
+		return do_extra_pass(env, prog);
 
 	return prog;
 }
diff --git a/arch/arm/net/bpf_jit_32.c b/arch/arm/net/bpf_jit_32.c
index e6b1bb2de627..1628b6fc70a4 100644
--- a/arch/arm/net/bpf_jit_32.c
+++ b/arch/arm/net/bpf_jit_32.c
@@ -2142,7 +2142,7 @@ bool bpf_jit_needs_zext(void)
 	return true;
 }
 
-struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
+struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog)
 {
 	struct bpf_binary_header *header;
 	struct jit_ctx ctx;
diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
index 566809be4a02..5c785eab4b5a 100644
--- a/arch/arm64/net/bpf_jit_comp.c
+++ b/arch/arm64/net/bpf_jit_comp.c
@@ -2013,7 +2013,7 @@ static void clear_jit_state(struct bpf_prog *prog)
 	prog->jited_len = 0;
 }
 
-struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
+struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog)
 {
 	int image_size, prog_size, extable_size, extable_align, extable_offset;
 	struct bpf_binary_header *header;
diff --git a/arch/loongarch/net/bpf_jit.c b/arch/loongarch/net/bpf_jit.c
index 57dd24d53c77..0ac69bac9d3c 100644
--- a/arch/loongarch/net/bpf_jit.c
+++ b/arch/loongarch/net/bpf_jit.c
@@ -1909,7 +1909,7 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
 	return ret < 0 ? ret : ret * LOONGARCH_INSN_SIZE;
 }
 
-struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
+struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog)
 {
 	bool extra_pass = false;
 	u8 *image_ptr, *ro_image_ptr;
diff --git a/arch/mips/net/bpf_jit_comp.c b/arch/mips/net/bpf_jit_comp.c
index d2b6c955f18e..6ee4abe6a1f7 100644
--- a/arch/mips/net/bpf_jit_comp.c
+++ b/arch/mips/net/bpf_jit_comp.c
@@ -909,7 +909,7 @@ bool bpf_jit_needs_zext(void)
 	return true;
 }
 
-struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
+struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog)
 {
 	struct bpf_binary_header *header = NULL;
 	struct jit_context ctx;
diff --git a/arch/parisc/net/bpf_jit_core.c b/arch/parisc/net/bpf_jit_core.c
index 4d339636a34a..90d081734d0f 100644
--- a/arch/parisc/net/bpf_jit_core.c
+++ b/arch/parisc/net/bpf_jit_core.c
@@ -41,7 +41,7 @@ bool bpf_jit_needs_zext(void)
 	return true;
 }
 
-struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
+struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog)
 {
 	unsigned int prog_size = 0, extable_size = 0;
 	bool extra_pass = false;
diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_comp.c
index 7a7c49640a2f..a6fd7ef249da 100644
--- a/arch/powerpc/net/bpf_jit_comp.c
+++ b/arch/powerpc/net/bpf_jit_comp.c
@@ -129,7 +129,7 @@ bool bpf_jit_needs_zext(void)
 	return true;
 }
 
-struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
+struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *fp)
 {
 	u32 proglen;
 	u32 alloclen;
diff --git a/arch/riscv/net/bpf_jit_core.c b/arch/riscv/net/bpf_jit_core.c
index c77e8aba14d3..bffaf584226c 100644
--- a/arch/riscv/net/bpf_jit_core.c
+++ b/arch/riscv/net/bpf_jit_core.c
@@ -41,7 +41,7 @@ bool bpf_jit_needs_zext(void)
 	return true;
 }
 
-struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
+struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog)
 {
 	unsigned int prog_size = 0, extable_size = 0;
 	bool extra_pass = false;
diff --git a/arch/s390/net/bpf_jit_comp.c b/arch/s390/net/bpf_jit_comp.c
index d6de2abfe4a7..061f16d1e3b1 100644
--- a/arch/s390/net/bpf_jit_comp.c
+++ b/arch/s390/net/bpf_jit_comp.c
@@ -2303,7 +2303,7 @@ static struct bpf_binary_header *bpf_jit_alloc(struct bpf_jit *jit,
 /*
  * Compile eBPF program "fp"
  */
-struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
+struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *fp)
 {
 	struct bpf_binary_header *header;
 	struct s390_jit_data *jit_data;
diff --git a/arch/sparc/net/bpf_jit_comp_64.c b/arch/sparc/net/bpf_jit_comp_64.c
index 86abd84d4005..eca9365c72e7 100644
--- a/arch/sparc/net/bpf_jit_comp_64.c
+++ b/arch/sparc/net/bpf_jit_comp_64.c
@@ -1477,7 +1477,7 @@ struct sparc64_jit_data {
 	struct jit_ctx ctx;
 };
 
-struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
+struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog)
 {
 	struct sparc64_jit_data *jit_data;
 	struct bpf_binary_header *header;
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index de51ab3a11ee..b95f23ad1093 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -3713,7 +3713,7 @@ struct x64_jit_data {
 #define MAX_PASSES 20
 #define PADDING_PASSES (MAX_PASSES - 5)
 
-struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
+struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog)
 {
 	struct bpf_binary_header *rw_header = NULL;
 	struct bpf_binary_header *header = NULL;
diff --git a/arch/x86/net/bpf_jit_comp32.c b/arch/x86/net/bpf_jit_comp32.c
index 5f259577614a..852baf2e4db4 100644
--- a/arch/x86/net/bpf_jit_comp32.c
+++ b/arch/x86/net/bpf_jit_comp32.c
@@ -2518,7 +2518,7 @@ bool bpf_jit_needs_zext(void)
 	return true;
 }
 
-struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
+struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog)
 {
 	struct bpf_binary_header *header = NULL;
 	int proglen, oldproglen = 0;
diff --git a/include/linux/filter.h b/include/linux/filter.h
index 2484d85be63d..d1bacfbbd02b 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -1108,6 +1108,8 @@ static inline int sk_filter_reason(struct sock *sk, struct sk_buff *skb,
 	return sk_filter_trim_cap(sk, skb, 1, reason);
 }
 
+struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct bpf_prog *fp,
+					   int *err);
 struct bpf_prog *bpf_prog_select_runtime(struct bpf_prog *fp, int *err);
 void bpf_prog_free(struct bpf_prog *fp);
 
@@ -1153,7 +1155,7 @@ u64 __bpf_call_base(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5);
 	((u64 (*)(u64, u64, u64, u64, u64, const struct bpf_insn *)) \
 	 (void *)__bpf_call_base)
 
-struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog);
+struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog);
 void bpf_jit_compile(struct bpf_prog *prog);
 bool bpf_jit_needs_zext(void);
 bool bpf_jit_inlines_helper_call(s32 imm);
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index c692213b1fdf..88a1834030a0 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -2432,18 +2432,8 @@ static bool bpf_prog_select_interpreter(struct bpf_prog *fp)
 	return select_interpreter;
 }
 
-/**
- *	bpf_prog_select_runtime - select exec runtime for BPF program
- *	@fp: bpf_prog populated with BPF program
- *	@err: pointer to error variable
- *
- * Try to JIT eBPF program, if JIT is not available, use interpreter.
- * The BPF program will be executed via bpf_prog_run() function.
- *
- * Return: the &fp argument along with &err set to 0 for success or
- * a negative errno code on failure
- */
-struct bpf_prog *bpf_prog_select_runtime(struct bpf_prog *fp, int *err)
+struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct bpf_prog *fp,
+					   int *err)
 {
 	/* In case of BPF to BPF calls, verifier did all the prep
 	 * work with regards to JITing, etc.
@@ -2471,7 +2461,7 @@ struct bpf_prog *bpf_prog_select_runtime(struct bpf_prog *fp, int *err)
 		if (*err)
 			return fp;
 
-		fp = bpf_int_jit_compile(fp);
+		fp = bpf_int_jit_compile(env, fp);
 		bpf_prog_jit_attempt_done(fp);
 		if (!fp->jited && jit_needed) {
 			*err = -ENOTSUPP;
@@ -2497,6 +2487,22 @@ struct bpf_prog *bpf_prog_select_runtime(struct bpf_prog *fp, int *err)
 
 	return fp;
 }
+
+/**
+ *	bpf_prog_select_runtime - select exec runtime for BPF program
+ *	@fp: bpf_prog populated with BPF program
+ *	@err: pointer to error variable
+ *
+ * Try to JIT eBPF program, if JIT is not available, use interpreter.
+ * The BPF program will be executed via bpf_prog_run() function.
+ *
+ * Return: the &fp argument along with &err set to 0 for success or
+ * a negative errno code on failure
+ */
+struct bpf_prog *bpf_prog_select_runtime(struct bpf_prog *fp, int *err)
+{
+	return __bpf_prog_select_runtime(NULL, fp, err);
+}
 EXPORT_SYMBOL_GPL(bpf_prog_select_runtime);
 
 static unsigned int __bpf_prog_ret1(const void *ctx,
@@ -2984,7 +2990,7 @@ const struct bpf_func_proto bpf_tail_call_proto = {
  * It is encouraged to implement bpf_int_jit_compile() instead, so that
  * eBPF and implicitly also cBPF can get JITed!
  */
-struct bpf_prog * __weak bpf_int_jit_compile(struct bpf_prog *prog)
+struct bpf_prog * __weak bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog)
 {
 	return prog;
 }
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 274039e36465..fb07be46c936 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -3090,10 +3090,6 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size)
 	if (err < 0)
 		goto free_used_maps;
 
-	prog = bpf_prog_select_runtime(prog, &err);
-	if (err < 0)
-		goto free_used_maps;
-
 	err = bpf_prog_mark_insn_arrays_ready(prog);
 	if (err < 0)
 		goto free_used_maps;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index e290c9b7d13d..fc3ba380b74a 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -22977,7 +22977,7 @@ static int jit_subprogs(struct bpf_verifier_env *env)
 		func[i]->aux->might_sleep = env->subprog_info[i].might_sleep;
 		if (!i)
 			func[i]->aux->exception_boundary = env->seen_exception;
-		func[i] = bpf_int_jit_compile(func[i]);
+		func[i] = bpf_int_jit_compile(env, func[i]);
 		if (!func[i]->jited) {
 			err = -ENOTSUPP;
 			goto out_free;
@@ -23021,7 +23021,7 @@ static int jit_subprogs(struct bpf_verifier_env *env)
 	}
 	for (i = 0; i < env->subprog_cnt; i++) {
 		old_bpf_func = func[i]->bpf_func;
-		tmp = bpf_int_jit_compile(func[i]);
+		tmp = bpf_int_jit_compile(env, func[i]);
 		if (tmp != func[i] || func[i]->bpf_func != old_bpf_func) {
 			verbose(env, "JIT doesn't support bpf-to-bpf calls\n");
 			err = -ENOTSUPP;
@@ -26227,6 +26227,10 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr, __u3
 		convert_pseudo_ld_imm64(env);
 	}
 
+	env->prog = __bpf_prog_select_runtime(env, env->prog, &ret);
+	if (ret)
+		goto err_release_maps;
+
 	adjust_btf_func(env);
 
 err_release_maps:
-- 
2.47.3



^ permalink raw reply related	[flat|nested] 20+ messages in thread

* Re: [bpf-next v8 2/5] bpf: Pass bpf_verifier_env to JIT
  2026-03-09 14:00 ` [bpf-next v8 2/5] bpf: Pass bpf_verifier_env to JIT Xu Kuohai
@ 2026-03-09 16:56   ` Anton Protopopov
  2026-03-10  6:44     ` Xu Kuohai
  0 siblings, 1 reply; 20+ messages in thread
From: Anton Protopopov @ 2026-03-09 16:56 UTC (permalink / raw)
  To: Xu Kuohai
  Cc: bpf, linux-kernel, linux-arm-kernel, Alexei Starovoitov,
	Daniel Borkmann, Andrii Nakryiko, Martin KaFai Lau,
	Eduard Zingerman, Yonghong Song, Puranjay Mohan, Shahab Vahedi,
	Russell King, Tiezhu Yang, Hengqi Chen, Johan Almbladh,
	Paul Burton, Hari Bathini, Christophe Leroy, Naveen N Rao,
	Luke Nelson, Xi Wang, Björn Töpel, Pu Lehui,
	Ilya Leoshkevich, Heiko Carstens, Vasily Gorbik, David S . Miller,
	Wang YanQing

On 26/03/09 10:00PM, Xu Kuohai wrote:
> From: Xu Kuohai <xukuohai@huawei.com>
> 
> Pass bpf_verifier_env to bpf_int_jit_compile(). The follow-up patch will
> use env->insn_aux_data in the JIT stage to detect indirect jump targets.
> 
> Since bpf_prog_select_runtime() can be called by cbpf and test code
> without verifier, introduce helper function __bpf_prog_select_runtime()
> to accept the env parameter.
> 
> Remove the call to bpf_prog_select_runtime() in bpf_prog_load(), and
> switch to call __bpf_prog_select_runtime() in the verifier, with env
> variable passed. The original bpf_prog_select_runtime() is preserved for
> cbpf and test code, where env is NULL.
> 
> Signed-off-by: Xu Kuohai <xukuohai@huawei.com>
> ---
>  arch/arc/net/bpf_jit_core.c      | 17 ++++++++--------
>  arch/arm/net/bpf_jit_32.c        |  2 +-
>  arch/arm64/net/bpf_jit_comp.c    |  2 +-
>  arch/loongarch/net/bpf_jit.c     |  2 +-
>  arch/mips/net/bpf_jit_comp.c     |  2 +-
>  arch/parisc/net/bpf_jit_core.c   |  2 +-
>  arch/powerpc/net/bpf_jit_comp.c  |  2 +-
>  arch/riscv/net/bpf_jit_core.c    |  2 +-
>  arch/s390/net/bpf_jit_comp.c     |  2 +-
>  arch/sparc/net/bpf_jit_comp_64.c |  2 +-
>  arch/x86/net/bpf_jit_comp.c      |  2 +-
>  arch/x86/net/bpf_jit_comp32.c    |  2 +-
>  include/linux/filter.h           |  4 +++-
>  kernel/bpf/core.c                | 34 +++++++++++++++++++-------------
>  kernel/bpf/syscall.c             |  4 ----
>  kernel/bpf/verifier.c            |  8 ++++++--
>  16 files changed, 49 insertions(+), 40 deletions(-)
> 
> diff --git a/arch/arc/net/bpf_jit_core.c b/arch/arc/net/bpf_jit_core.c
> index 12facf5750da..cf708c308adb 100644
> --- a/arch/arc/net/bpf_jit_core.c
> +++ b/arch/arc/net/bpf_jit_core.c
> @@ -153,7 +153,8 @@ static void jit_dump(const struct jit_context *ctx)
>  }
>  
>  /* Initialise the context so there's no garbage. */
> -static int jit_ctx_init(struct jit_context *ctx, struct bpf_prog *prog)
> +static int jit_ctx_init(struct jit_context *ctx, struct bpf_verifier_env *env,
> +			struct bpf_prog *prog)
>  {
>  	memset(ctx, 0, sizeof(*ctx));
>  
> @@ -1317,7 +1318,7 @@ static int jit_patch_relocations(struct jit_context *ctx)
>   * to get the necessary data for the real compilation phase,
>   * jit_compile().
>   */
> -static struct bpf_prog *do_normal_pass(struct bpf_prog *prog)
> +static struct bpf_prog *do_normal_pass(struct bpf_verifier_env *env, struct bpf_prog *prog)
>  {
>  	struct jit_context ctx;
>  
> @@ -1325,7 +1326,7 @@ static struct bpf_prog *do_normal_pass(struct bpf_prog *prog)
>  	if (!prog->jit_requested)
>  		return prog;
>  
> -	if (jit_ctx_init(&ctx, prog)) {
> +	if (jit_ctx_init(&ctx, env, prog)) {
>  		jit_ctx_cleanup(&ctx);
>  		return prog;
>  	}
> @@ -1356,7 +1357,7 @@ static struct bpf_prog *do_normal_pass(struct bpf_prog *prog)
>   * again to get the newly translated addresses in order to resolve
>   * the "call"s.
>   */
> -static struct bpf_prog *do_extra_pass(struct bpf_prog *prog)
> +static struct bpf_prog *do_extra_pass(struct bpf_verifier_env *env, struct bpf_prog *prog)
>  {
>  	struct jit_context ctx;
>  
> @@ -1364,7 +1365,7 @@ static struct bpf_prog *do_extra_pass(struct bpf_prog *prog)
>  	if (check_jit_context(prog))
>  		return prog;
>  
> -	if (jit_ctx_init(&ctx, prog)) {
> +	if (jit_ctx_init(&ctx, env, prog)) {
>  		jit_ctx_cleanup(&ctx);
>  		return prog;
>  	}
> @@ -1393,15 +1394,15 @@ static struct bpf_prog *do_extra_pass(struct bpf_prog *prog)
>   * (re)locations involved that their addresses are not known
>   * during the first run.
>   */
> -struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
> +struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog)
>  {
>  	vm_dump(prog);
>  
>  	/* Was this program already translated? */
>  	if (!prog->jited)
> -		return do_normal_pass(prog);
> +		return do_normal_pass(env, prog);
>  	else
> -		return do_extra_pass(prog);
> +		return do_extra_pass(env, prog);

Why is this necessary to pass env all the way through jit_ctx_init?
All below looks good, so minus the above:

Reviewed-by: Anton Protopopov <a.s.protopopov@gmail.com>

>  
>  	return prog;
>  }
> diff --git a/arch/arm/net/bpf_jit_32.c b/arch/arm/net/bpf_jit_32.c
> index e6b1bb2de627..1628b6fc70a4 100644
> --- a/arch/arm/net/bpf_jit_32.c
> +++ b/arch/arm/net/bpf_jit_32.c
> @@ -2142,7 +2142,7 @@ bool bpf_jit_needs_zext(void)
>  	return true;
>  }
>  
> -struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
> +struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog)
>  {
>  	struct bpf_binary_header *header;
>  	struct jit_ctx ctx;
> diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
> index 566809be4a02..5c785eab4b5a 100644
> --- a/arch/arm64/net/bpf_jit_comp.c
> +++ b/arch/arm64/net/bpf_jit_comp.c
> @@ -2013,7 +2013,7 @@ static void clear_jit_state(struct bpf_prog *prog)
>  	prog->jited_len = 0;
>  }
>  
> -struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
> +struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog)
>  {
>  	int image_size, prog_size, extable_size, extable_align, extable_offset;
>  	struct bpf_binary_header *header;
> diff --git a/arch/loongarch/net/bpf_jit.c b/arch/loongarch/net/bpf_jit.c
> index 57dd24d53c77..0ac69bac9d3c 100644
> --- a/arch/loongarch/net/bpf_jit.c
> +++ b/arch/loongarch/net/bpf_jit.c
> @@ -1909,7 +1909,7 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
>  	return ret < 0 ? ret : ret * LOONGARCH_INSN_SIZE;
>  }
>  
> -struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
> +struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog)
>  {
>  	bool extra_pass = false;
>  	u8 *image_ptr, *ro_image_ptr;
> diff --git a/arch/mips/net/bpf_jit_comp.c b/arch/mips/net/bpf_jit_comp.c
> index d2b6c955f18e..6ee4abe6a1f7 100644
> --- a/arch/mips/net/bpf_jit_comp.c
> +++ b/arch/mips/net/bpf_jit_comp.c
> @@ -909,7 +909,7 @@ bool bpf_jit_needs_zext(void)
>  	return true;
>  }
>  
> -struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
> +struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog)
>  {
>  	struct bpf_binary_header *header = NULL;
>  	struct jit_context ctx;
> diff --git a/arch/parisc/net/bpf_jit_core.c b/arch/parisc/net/bpf_jit_core.c
> index 4d339636a34a..90d081734d0f 100644
> --- a/arch/parisc/net/bpf_jit_core.c
> +++ b/arch/parisc/net/bpf_jit_core.c
> @@ -41,7 +41,7 @@ bool bpf_jit_needs_zext(void)
>  	return true;
>  }
>  
> -struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
> +struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog)
>  {
>  	unsigned int prog_size = 0, extable_size = 0;
>  	bool extra_pass = false;
> diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_comp.c
> index 7a7c49640a2f..a6fd7ef249da 100644
> --- a/arch/powerpc/net/bpf_jit_comp.c
> +++ b/arch/powerpc/net/bpf_jit_comp.c
> @@ -129,7 +129,7 @@ bool bpf_jit_needs_zext(void)
>  	return true;
>  }
>  
> -struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
> +struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *fp)
>  {
>  	u32 proglen;
>  	u32 alloclen;
> diff --git a/arch/riscv/net/bpf_jit_core.c b/arch/riscv/net/bpf_jit_core.c
> index c77e8aba14d3..bffaf584226c 100644
> --- a/arch/riscv/net/bpf_jit_core.c
> +++ b/arch/riscv/net/bpf_jit_core.c
> @@ -41,7 +41,7 @@ bool bpf_jit_needs_zext(void)
>  	return true;
>  }
>  
> -struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
> +struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog)
>  {
>  	unsigned int prog_size = 0, extable_size = 0;
>  	bool extra_pass = false;
> diff --git a/arch/s390/net/bpf_jit_comp.c b/arch/s390/net/bpf_jit_comp.c
> index d6de2abfe4a7..061f16d1e3b1 100644
> --- a/arch/s390/net/bpf_jit_comp.c
> +++ b/arch/s390/net/bpf_jit_comp.c
> @@ -2303,7 +2303,7 @@ static struct bpf_binary_header *bpf_jit_alloc(struct bpf_jit *jit,
>  /*
>   * Compile eBPF program "fp"
>   */
> -struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
> +struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *fp)
>  {
>  	struct bpf_binary_header *header;
>  	struct s390_jit_data *jit_data;
> diff --git a/arch/sparc/net/bpf_jit_comp_64.c b/arch/sparc/net/bpf_jit_comp_64.c
> index 86abd84d4005..eca9365c72e7 100644
> --- a/arch/sparc/net/bpf_jit_comp_64.c
> +++ b/arch/sparc/net/bpf_jit_comp_64.c
> @@ -1477,7 +1477,7 @@ struct sparc64_jit_data {
>  	struct jit_ctx ctx;
>  };
>  
> -struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
> +struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog)
>  {
>  	struct sparc64_jit_data *jit_data;
>  	struct bpf_binary_header *header;
> diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
> index de51ab3a11ee..b95f23ad1093 100644
> --- a/arch/x86/net/bpf_jit_comp.c
> +++ b/arch/x86/net/bpf_jit_comp.c
> @@ -3713,7 +3713,7 @@ struct x64_jit_data {
>  #define MAX_PASSES 20
>  #define PADDING_PASSES (MAX_PASSES - 5)
>  
> -struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
> +struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog)
>  {
>  	struct bpf_binary_header *rw_header = NULL;
>  	struct bpf_binary_header *header = NULL;
> diff --git a/arch/x86/net/bpf_jit_comp32.c b/arch/x86/net/bpf_jit_comp32.c
> index 5f259577614a..852baf2e4db4 100644
> --- a/arch/x86/net/bpf_jit_comp32.c
> +++ b/arch/x86/net/bpf_jit_comp32.c
> @@ -2518,7 +2518,7 @@ bool bpf_jit_needs_zext(void)
>  	return true;
>  }
>  
> -struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
> +struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog)
>  {
>  	struct bpf_binary_header *header = NULL;
>  	int proglen, oldproglen = 0;
> diff --git a/include/linux/filter.h b/include/linux/filter.h
> index 2484d85be63d..d1bacfbbd02b 100644
> --- a/include/linux/filter.h
> +++ b/include/linux/filter.h
> @@ -1108,6 +1108,8 @@ static inline int sk_filter_reason(struct sock *sk, struct sk_buff *skb,
>  	return sk_filter_trim_cap(sk, skb, 1, reason);
>  }
>  
> +struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct bpf_prog *fp,
> +					   int *err);
>  struct bpf_prog *bpf_prog_select_runtime(struct bpf_prog *fp, int *err);
>  void bpf_prog_free(struct bpf_prog *fp);
>  
> @@ -1153,7 +1155,7 @@ u64 __bpf_call_base(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5);
>  	((u64 (*)(u64, u64, u64, u64, u64, const struct bpf_insn *)) \
>  	 (void *)__bpf_call_base)
>  
> -struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog);
> +struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog);
>  void bpf_jit_compile(struct bpf_prog *prog);
>  bool bpf_jit_needs_zext(void);
>  bool bpf_jit_inlines_helper_call(s32 imm);
> diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
> index c692213b1fdf..88a1834030a0 100644
> --- a/kernel/bpf/core.c
> +++ b/kernel/bpf/core.c
> @@ -2432,18 +2432,8 @@ static bool bpf_prog_select_interpreter(struct bpf_prog *fp)
>  	return select_interpreter;
>  }
>  
> -/**
> - *	bpf_prog_select_runtime - select exec runtime for BPF program
> - *	@fp: bpf_prog populated with BPF program
> - *	@err: pointer to error variable
> - *
> - * Try to JIT eBPF program, if JIT is not available, use interpreter.
> - * The BPF program will be executed via bpf_prog_run() function.
> - *
> - * Return: the &fp argument along with &err set to 0 for success or
> - * a negative errno code on failure
> - */
> -struct bpf_prog *bpf_prog_select_runtime(struct bpf_prog *fp, int *err)
> +struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct bpf_prog *fp,
> +					   int *err)
>  {
>  	/* In case of BPF to BPF calls, verifier did all the prep
>  	 * work with regards to JITing, etc.
> @@ -2471,7 +2461,7 @@ struct bpf_prog *bpf_prog_select_runtime(struct bpf_prog *fp, int *err)
>  		if (*err)
>  			return fp;
>  
> -		fp = bpf_int_jit_compile(fp);
> +		fp = bpf_int_jit_compile(env, fp);
>  		bpf_prog_jit_attempt_done(fp);
>  		if (!fp->jited && jit_needed) {
>  			*err = -ENOTSUPP;
> @@ -2497,6 +2487,22 @@ struct bpf_prog *bpf_prog_select_runtime(struct bpf_prog *fp, int *err)
>  
>  	return fp;
>  }
> +
> +/**
> + *	bpf_prog_select_runtime - select exec runtime for BPF program
> + *	@fp: bpf_prog populated with BPF program
> + *	@err: pointer to error variable
> + *
> + * Try to JIT eBPF program, if JIT is not available, use interpreter.
> + * The BPF program will be executed via bpf_prog_run() function.
> + *
> + * Return: the &fp argument along with &err set to 0 for success or
> + * a negative errno code on failure
> + */
> +struct bpf_prog *bpf_prog_select_runtime(struct bpf_prog *fp, int *err)
> +{
> +	return __bpf_prog_select_runtime(NULL, fp, err);
> +}
>  EXPORT_SYMBOL_GPL(bpf_prog_select_runtime);
>  
>  static unsigned int __bpf_prog_ret1(const void *ctx,
> @@ -2984,7 +2990,7 @@ const struct bpf_func_proto bpf_tail_call_proto = {
>   * It is encouraged to implement bpf_int_jit_compile() instead, so that
>   * eBPF and implicitly also cBPF can get JITed!
>   */
> -struct bpf_prog * __weak bpf_int_jit_compile(struct bpf_prog *prog)
> +struct bpf_prog * __weak bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog)
>  {
>  	return prog;
>  }
> diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
> index 274039e36465..fb07be46c936 100644
> --- a/kernel/bpf/syscall.c
> +++ b/kernel/bpf/syscall.c
> @@ -3090,10 +3090,6 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size)
>  	if (err < 0)
>  		goto free_used_maps;
>  
> -	prog = bpf_prog_select_runtime(prog, &err);
> -	if (err < 0)
> -		goto free_used_maps;
> -
>  	err = bpf_prog_mark_insn_arrays_ready(prog);
>  	if (err < 0)
>  		goto free_used_maps;
> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index e290c9b7d13d..fc3ba380b74a 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
> @@ -22977,7 +22977,7 @@ static int jit_subprogs(struct bpf_verifier_env *env)
>  		func[i]->aux->might_sleep = env->subprog_info[i].might_sleep;
>  		if (!i)
>  			func[i]->aux->exception_boundary = env->seen_exception;
> -		func[i] = bpf_int_jit_compile(func[i]);
> +		func[i] = bpf_int_jit_compile(env, func[i]);
>  		if (!func[i]->jited) {
>  			err = -ENOTSUPP;
>  			goto out_free;
> @@ -23021,7 +23021,7 @@ static int jit_subprogs(struct bpf_verifier_env *env)
>  	}
>  	for (i = 0; i < env->subprog_cnt; i++) {
>  		old_bpf_func = func[i]->bpf_func;
> -		tmp = bpf_int_jit_compile(func[i]);
> +		tmp = bpf_int_jit_compile(env, func[i]);
>  		if (tmp != func[i] || func[i]->bpf_func != old_bpf_func) {
>  			verbose(env, "JIT doesn't support bpf-to-bpf calls\n");
>  			err = -ENOTSUPP;
> @@ -26227,6 +26227,10 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr, __u3
>  		convert_pseudo_ld_imm64(env);
>  	}
>  
> +	env->prog = __bpf_prog_select_runtime(env, env->prog, &ret);
> +	if (ret)
> +		goto err_release_maps;
> +
>  	adjust_btf_func(env);
>  
>  err_release_maps:
> -- 
> 2.47.3
> 


^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [bpf-next v8 2/5] bpf: Pass bpf_verifier_env to JIT
  2026-03-09 16:56   ` Anton Protopopov
@ 2026-03-10  6:44     ` Xu Kuohai
  0 siblings, 0 replies; 20+ messages in thread
From: Xu Kuohai @ 2026-03-10  6:44 UTC (permalink / raw)
  To: Anton Protopopov
  Cc: bpf, linux-kernel, linux-arm-kernel, Alexei Starovoitov,
	Daniel Borkmann, Andrii Nakryiko, Martin KaFai Lau,
	Eduard Zingerman, Yonghong Song, Puranjay Mohan, Shahab Vahedi,
	Russell King, Tiezhu Yang, Hengqi Chen, Johan Almbladh,
	Paul Burton, Hari Bathini, Christophe Leroy, Naveen N Rao,
	Luke Nelson, Xi Wang, Björn Töpel, Pu Lehui,
	Ilya Leoshkevich, Heiko Carstens, Vasily Gorbik, David S . Miller,
	Wang YanQing

On 3/10/2026 12:56 AM, Anton Protopopov wrote:

[...]

>> -struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>> +struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog)
>>   {
>>   	vm_dump(prog);
>>   
>>   	/* Was this program already translated? */
>>   	if (!prog->jited)
>> -		return do_normal_pass(prog);
>> +		return do_normal_pass(env, prog);
>>   	else
>> -		return do_extra_pass(prog);
>> +		return do_extra_pass(env, prog);
> 
> Why is this necessary to pass env all the way through jit_ctx_init?


Right, this is indeed stale code. In the earlier version, jit_ctx_init
invoked constants blinding function which required env. But constants
blinding was removed, passing env is not necessary now.

> All below looks good, so minus the above:
> 
> Reviewed-by: Anton Protopopov <a.s.protopopov@gmail.com>

Thanks for the reviewing!



^ permalink raw reply	[flat|nested] 20+ messages in thread

* [bpf-next v8 3/5] bpf: Add helper to detect indirect jump targets
  2026-03-09 14:00 [bpf-next v8 0/5] emit ENDBR/BTI instructions for indirect jump targets Xu Kuohai
  2026-03-09 14:00 ` [bpf-next v8 1/5] bpf: Move constants blinding from JIT to verifier Xu Kuohai
  2026-03-09 14:00 ` [bpf-next v8 2/5] bpf: Pass bpf_verifier_env to JIT Xu Kuohai
@ 2026-03-09 14:00 ` Xu Kuohai
  2026-03-09 17:30   ` Anton Protopopov
  2026-03-09 14:00 ` [bpf-next v8 4/5] bpf, x86: Emit ENDBR for " Xu Kuohai
                   ` (3 subsequent siblings)
  6 siblings, 1 reply; 20+ messages in thread
From: Xu Kuohai @ 2026-03-09 14:00 UTC (permalink / raw)
  To: bpf, linux-kernel, linux-arm-kernel
  Cc: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
	Martin KaFai Lau, Eduard Zingerman, Yonghong Song, Puranjay Mohan,
	Anton Protopopov, Shahab Vahedi, Russell King, Tiezhu Yang,
	Hengqi Chen, Johan Almbladh, Paul Burton, Hari Bathini,
	Christophe Leroy, Naveen N Rao, Luke Nelson, Xi Wang,
	Björn Töpel, Pu Lehui, Ilya Leoshkevich, Heiko Carstens,
	Vasily Gorbik, David S . Miller, Wang YanQing

From: Xu Kuohai <xukuohai@huawei.com>

Introduce helper bpf_insn_is_indirect_target to check whether a BPF
instruction is an indirect jump target.

Since the verifier knows which instructions are indirect jump targets,
add a new flag indirect_target to struct bpf_insn_aux_data to mark
them. The verifier sets this flag when verifing an indirect jump target
instruction, and the helper checks it to determine whether an
instruction is an indirect jump target.

Signed-off-by: Xu Kuohai <xukuohai@huawei.com>
---
 include/linux/bpf.h          |  2 ++
 include/linux/bpf_verifier.h |  9 +++++----
 kernel/bpf/core.c            |  9 +++++++++
 kernel/bpf/verifier.c        | 18 ++++++++++++++++++
 4 files changed, 34 insertions(+), 4 deletions(-)

diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 05b34a6355b0..90760e250865 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -1541,6 +1541,8 @@ bool bpf_has_frame_pointer(unsigned long ip);
 int bpf_jit_charge_modmem(u32 size);
 void bpf_jit_uncharge_modmem(u32 size);
 bool bpf_prog_has_trampoline(const struct bpf_prog *prog);
+bool bpf_insn_is_indirect_target(const struct bpf_verifier_env *env, const struct bpf_prog *prog,
+				 int insn_idx);
 #else
 static inline int bpf_trampoline_link_prog(struct bpf_tramp_link *link,
 					   struct bpf_trampoline *tr,
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 090aa26d1c98..6431b94746cf 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -578,16 +578,17 @@ struct bpf_insn_aux_data {
 
 	/* below fields are initialized once */
 	unsigned int orig_idx; /* original instruction index */
-	bool jmp_point;
-	bool prune_point;
+	u32 jmp_point:1;
+	u32 prune_point:1;
 	/* ensure we check state equivalence and save state checkpoint and
 	 * this instruction, regardless of any heuristics
 	 */
-	bool force_checkpoint;
+	u32 force_checkpoint:1;
 	/* true if instruction is a call to a helper function that
 	 * accepts callback function as a parameter.
 	 */
-	bool calls_callback;
+	u32 calls_callback:1;
+	u32 indirect_target:1; /* if it is an indirect jump target */
 	/*
 	 * CFG strongly connected component this instruction belongs to,
 	 * zero if it is a singleton SCC.
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 88a1834030a0..ccc618a50103 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -1480,6 +1480,15 @@ int bpf_jit_blind_constants(struct bpf_verifier_env *env)
 	prog->blinded = 1;
 	return 0;
 }
+
+bool bpf_insn_is_indirect_target(const struct bpf_verifier_env *env, const struct bpf_prog *prog,
+				 int insn_idx)
+{
+	if (!env)
+		return false;
+	insn_idx += prog->aux->subprog_start;
+	return env->insn_aux_data[insn_idx].indirect_target;
+}
 #endif /* CONFIG_BPF_JIT */
 
 /* Base function for offset calculation. Needs to go into .text section,
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index fc3ba380b74a..8d8e349f98cd 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -4022,6 +4022,11 @@ static bool is_jmp_point(struct bpf_verifier_env *env, int insn_idx)
 	return env->insn_aux_data[insn_idx].jmp_point;
 }
 
+static void mark_indirect_target(struct bpf_verifier_env *env, int idx)
+{
+	env->insn_aux_data[idx].indirect_target = true;
+}
+
 #define LR_FRAMENO_BITS	3
 #define LR_SPI_BITS	6
 #define LR_ENTRY_BITS	(LR_SPI_BITS + LR_FRAMENO_BITS + 1)
@@ -21082,12 +21087,14 @@ static int check_indirect_jump(struct bpf_verifier_env *env, struct bpf_insn *in
 	}
 
 	for (i = 0; i < n - 1; i++) {
+		mark_indirect_target(env, env->gotox_tmp_buf->items[i]);
 		other_branch = push_stack(env, env->gotox_tmp_buf->items[i],
 					  env->insn_idx, env->cur_state->speculative);
 		if (IS_ERR(other_branch))
 			return PTR_ERR(other_branch);
 	}
 	env->insn_idx = env->gotox_tmp_buf->items[n-1];
+	mark_indirect_target(env, env->insn_idx);
 	return 0;
 }
 
@@ -22013,6 +22020,17 @@ static void adjust_insn_aux_data(struct bpf_verifier_env *env,
 		data[i].seen = old_seen;
 		data[i].zext_dst = insn_has_def32(insn + i);
 	}
+
+	/* The indirect_target flag of the original instruction was moved to the last of the
+	 * new instructions by the above memmove and memset, but the indirect jump target is
+	 * actually the first instruction, so move it back. This also matches with the behavior
+	 * of bpf_insn_array_adjust(), which preserves xlated_off to point to the first new
+	 * instruction.
+	 */
+	if (data[off + cnt - 1].indirect_target) {
+		data[off].indirect_target = 1;
+		data[off + cnt - 1].indirect_target = 0;
+	}
 }
 
 static void adjust_subprog_starts(struct bpf_verifier_env *env, u32 off, u32 len)
-- 
2.47.3



^ permalink raw reply related	[flat|nested] 20+ messages in thread

* Re: [bpf-next v8 3/5] bpf: Add helper to detect indirect jump targets
  2026-03-09 14:00 ` [bpf-next v8 3/5] bpf: Add helper to detect indirect jump targets Xu Kuohai
@ 2026-03-09 17:30   ` Anton Protopopov
  0 siblings, 0 replies; 20+ messages in thread
From: Anton Protopopov @ 2026-03-09 17:30 UTC (permalink / raw)
  To: Xu Kuohai
  Cc: bpf, linux-kernel, linux-arm-kernel, Alexei Starovoitov,
	Daniel Borkmann, Andrii Nakryiko, Martin KaFai Lau,
	Eduard Zingerman, Yonghong Song, Puranjay Mohan, Shahab Vahedi,
	Russell King, Tiezhu Yang, Hengqi Chen, Johan Almbladh,
	Paul Burton, Hari Bathini, Christophe Leroy, Naveen N Rao,
	Luke Nelson, Xi Wang, Björn Töpel, Pu Lehui,
	Ilya Leoshkevich, Heiko Carstens, Vasily Gorbik, David S . Miller,
	Wang YanQing

On 26/03/09 10:00PM, Xu Kuohai wrote:
> From: Xu Kuohai <xukuohai@huawei.com>
> 
> Introduce helper bpf_insn_is_indirect_target to check whether a BPF
> instruction is an indirect jump target.
> 
> Since the verifier knows which instructions are indirect jump targets,
> add a new flag indirect_target to struct bpf_insn_aux_data to mark
> them. The verifier sets this flag when verifing an indirect jump target
> instruction, and the helper checks it to determine whether an
> instruction is an indirect jump target.
> 
> Signed-off-by: Xu Kuohai <xukuohai@huawei.com>
> ---
>  include/linux/bpf.h          |  2 ++
>  include/linux/bpf_verifier.h |  9 +++++----
>  kernel/bpf/core.c            |  9 +++++++++
>  kernel/bpf/verifier.c        | 18 ++++++++++++++++++
>  4 files changed, 34 insertions(+), 4 deletions(-)
> 
> diff --git a/include/linux/bpf.h b/include/linux/bpf.h
> index 05b34a6355b0..90760e250865 100644
> --- a/include/linux/bpf.h
> +++ b/include/linux/bpf.h
> @@ -1541,6 +1541,8 @@ bool bpf_has_frame_pointer(unsigned long ip);
>  int bpf_jit_charge_modmem(u32 size);
>  void bpf_jit_uncharge_modmem(u32 size);
>  bool bpf_prog_has_trampoline(const struct bpf_prog *prog);
> +bool bpf_insn_is_indirect_target(const struct bpf_verifier_env *env, const struct bpf_prog *prog,
> +				 int insn_idx);
>  #else
>  static inline int bpf_trampoline_link_prog(struct bpf_tramp_link *link,
>  					   struct bpf_trampoline *tr,
> diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
> index 090aa26d1c98..6431b94746cf 100644
> --- a/include/linux/bpf_verifier.h
> +++ b/include/linux/bpf_verifier.h
> @@ -578,16 +578,17 @@ struct bpf_insn_aux_data {
>  
>  	/* below fields are initialized once */
>  	unsigned int orig_idx; /* original instruction index */
> -	bool jmp_point;
> -	bool prune_point;
> +	u32 jmp_point:1;
> +	u32 prune_point:1;
>  	/* ensure we check state equivalence and save state checkpoint and
>  	 * this instruction, regardless of any heuristics
>  	 */
> -	bool force_checkpoint;
> +	u32 force_checkpoint:1;
>  	/* true if instruction is a call to a helper function that
>  	 * accepts callback function as a parameter.
>  	 */
> -	bool calls_callback;
> +	u32 calls_callback:1;
> +	u32 indirect_target:1; /* if it is an indirect jump target */
>  	/*
>  	 * CFG strongly connected component this instruction belongs to,
>  	 * zero if it is a singleton SCC.
> diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
> index 88a1834030a0..ccc618a50103 100644
> --- a/kernel/bpf/core.c
> +++ b/kernel/bpf/core.c
> @@ -1480,6 +1480,15 @@ int bpf_jit_blind_constants(struct bpf_verifier_env *env)
>  	prog->blinded = 1;
>  	return 0;
>  }
> +
> +bool bpf_insn_is_indirect_target(const struct bpf_verifier_env *env, const struct bpf_prog *prog,
> +				 int insn_idx)
> +{
> +	if (!env)
> +		return false;
> +	insn_idx += prog->aux->subprog_start;
> +	return env->insn_aux_data[insn_idx].indirect_target;
> +}
>  #endif /* CONFIG_BPF_JIT */
>  
>  /* Base function for offset calculation. Needs to go into .text section,
> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index fc3ba380b74a..8d8e349f98cd 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
> @@ -4022,6 +4022,11 @@ static bool is_jmp_point(struct bpf_verifier_env *env, int insn_idx)
>  	return env->insn_aux_data[insn_idx].jmp_point;
>  }
>  
> +static void mark_indirect_target(struct bpf_verifier_env *env, int idx)
> +{
> +	env->insn_aux_data[idx].indirect_target = true;
> +}
> +
>  #define LR_FRAMENO_BITS	3
>  #define LR_SPI_BITS	6
>  #define LR_ENTRY_BITS	(LR_SPI_BITS + LR_FRAMENO_BITS + 1)
> @@ -21082,12 +21087,14 @@ static int check_indirect_jump(struct bpf_verifier_env *env, struct bpf_insn *in
>  	}
>  
>  	for (i = 0; i < n - 1; i++) {
> +		mark_indirect_target(env, env->gotox_tmp_buf->items[i]);
>  		other_branch = push_stack(env, env->gotox_tmp_buf->items[i],
>  					  env->insn_idx, env->cur_state->speculative);
>  		if (IS_ERR(other_branch))
>  			return PTR_ERR(other_branch);
>  	}
>  	env->insn_idx = env->gotox_tmp_buf->items[n-1];
> +	mark_indirect_target(env, env->insn_idx);
>  	return 0;
>  }
>  
> @@ -22013,6 +22020,17 @@ static void adjust_insn_aux_data(struct bpf_verifier_env *env,
>  		data[i].seen = old_seen;
>  		data[i].zext_dst = insn_has_def32(insn + i);
>  	}
> +
> +	/* The indirect_target flag of the original instruction was moved to the last of the
> +	 * new instructions by the above memmove and memset, but the indirect jump target is
> +	 * actually the first instruction, so move it back. This also matches with the behavior
> +	 * of bpf_insn_array_adjust(), which preserves xlated_off to point to the first new
> +	 * instruction.
> +	 */
> +	if (data[off + cnt - 1].indirect_target) {
> +		data[off].indirect_target = 1;
> +		data[off + cnt - 1].indirect_target = 0;
> +	}
>  }
>  
>  static void adjust_subprog_starts(struct bpf_verifier_env *env, u32 off, u32 len)
> -- 
> 2.47.3
> 

Reviewed-by: Anton Protopopov <a.s.protopopov@gmail.com>


^ permalink raw reply	[flat|nested] 20+ messages in thread

* [bpf-next v8 4/5] bpf, x86: Emit ENDBR for indirect jump targets
  2026-03-09 14:00 [bpf-next v8 0/5] emit ENDBR/BTI instructions for indirect jump targets Xu Kuohai
                   ` (2 preceding siblings ...)
  2026-03-09 14:00 ` [bpf-next v8 3/5] bpf: Add helper to detect indirect jump targets Xu Kuohai
@ 2026-03-09 14:00 ` Xu Kuohai
  2026-03-09 16:37   ` Anton Protopopov
  2026-03-09 14:00 ` [bpf-next v8 5/5] bpf, arm64: Emit BTI for indirect jump target Xu Kuohai
                   ` (2 subsequent siblings)
  6 siblings, 1 reply; 20+ messages in thread
From: Xu Kuohai @ 2026-03-09 14:00 UTC (permalink / raw)
  To: bpf, linux-kernel, linux-arm-kernel
  Cc: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
	Martin KaFai Lau, Eduard Zingerman, Yonghong Song, Puranjay Mohan,
	Anton Protopopov, Shahab Vahedi, Russell King, Tiezhu Yang,
	Hengqi Chen, Johan Almbladh, Paul Burton, Hari Bathini,
	Christophe Leroy, Naveen N Rao, Luke Nelson, Xi Wang,
	Björn Töpel, Pu Lehui, Ilya Leoshkevich, Heiko Carstens,
	Vasily Gorbik, David S . Miller, Wang YanQing

From: Xu Kuohai <xukuohai@huawei.com>

On CPUs that support CET/IBT, the indirect jump selftest triggers
a kernel panic because the indirect jump targets lack ENDBR
instructions.

To fix it, emit an ENDBR instruction to each indirect jump target. Since
the ENDBR instruction shifts the position of original jited instructions,
fix the instruction address calculation wherever the addresses are used.

For reference, below is a sample panic log.

 Missing ENDBR: bpf_prog_2e5f1c71c13ac3e0_big_jump_table+0x97/0xe1
 ------------[ cut here ]------------
 kernel BUG at arch/x86/kernel/cet.c:133!
 Oops: invalid opcode: 0000 [#1] SMP NOPTI

 ...

  ? 0xffffffffc00fb258
  ? bpf_prog_2e5f1c71c13ac3e0_big_jump_table+0x97/0xe1
  bpf_prog_test_run_syscall+0x110/0x2f0
  ? fdget+0xba/0xe0
  __sys_bpf+0xe4b/0x2590
  ? __kmalloc_node_track_caller_noprof+0x1c7/0x680
  ? bpf_prog_test_run_syscall+0x215/0x2f0
  __x64_sys_bpf+0x21/0x30
  do_syscall_64+0x85/0x620
  ? bpf_prog_test_run_syscall+0x1e2/0x2f0

Fixes: 493d9e0d6083 ("bpf, x86: add support for indirect jumps")
Signed-off-by: Xu Kuohai <xukuohai@huawei.com>
---
 arch/x86/net/bpf_jit_comp.c | 26 +++++++++++++++-----------
 1 file changed, 15 insertions(+), 11 deletions(-)

diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index b95f23ad1093..251dff1cd8e4 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -1649,8 +1649,8 @@ static int emit_spectre_bhb_barrier(u8 **pprog, u8 *ip,
 	return 0;
 }
 
-static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image,
-		  int oldproglen, struct jit_context *ctx, bool jmp_padding)
+static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *addrs, u8 *image,
+		  u8 *rw_image, int oldproglen, struct jit_context *ctx, bool jmp_padding)
 {
 	bool tail_call_reachable = bpf_prog->aux->tail_call_reachable;
 	struct bpf_insn *insn = bpf_prog->insnsi;
@@ -1663,7 +1663,7 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image
 	void __percpu *priv_stack_ptr;
 	int i, excnt = 0;
 	int ilen, proglen = 0;
-	u8 *prog = temp;
+	u8 *ip, *prog = temp;
 	u32 stack_depth;
 	int err;
 
@@ -1734,6 +1734,13 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image
 				dst_reg = X86_REG_R9;
 		}
 
+#ifdef CONFIG_X86_KERNEL_IBT
+		if (bpf_insn_is_indirect_target(env, bpf_prog, i - 1))
+			EMIT_ENDBR();
+#endif
+
+		ip = image + addrs[i - 1] + (prog - temp);
+
 		switch (insn->code) {
 			/* ALU */
 		case BPF_ALU | BPF_ADD | BPF_X:
@@ -2440,8 +2447,6 @@ st:			if (is_imm8(insn->off))
 
 			/* call */
 		case BPF_JMP | BPF_CALL: {
-			u8 *ip = image + addrs[i - 1];
-
 			func = (u8 *) __bpf_call_base + imm32;
 			if (src_reg == BPF_PSEUDO_CALL && tail_call_reachable) {
 				LOAD_TAIL_CALL_CNT_PTR(stack_depth);
@@ -2465,7 +2470,8 @@ st:			if (is_imm8(insn->off))
 			if (imm32)
 				emit_bpf_tail_call_direct(bpf_prog,
 							  &bpf_prog->aux->poke_tab[imm32 - 1],
-							  &prog, image + addrs[i - 1],
+							  &prog,
+							  ip,
 							  callee_regs_used,
 							  stack_depth,
 							  ctx);
@@ -2474,7 +2480,7 @@ st:			if (is_imm8(insn->off))
 							    &prog,
 							    callee_regs_used,
 							    stack_depth,
-							    image + addrs[i - 1],
+							    ip,
 							    ctx);
 			break;
 
@@ -2639,7 +2645,7 @@ st:			if (is_imm8(insn->off))
 			break;
 
 		case BPF_JMP | BPF_JA | BPF_X:
-			emit_indirect_jump(&prog, insn->dst_reg, image + addrs[i - 1]);
+			emit_indirect_jump(&prog, insn->dst_reg, ip);
 			break;
 		case BPF_JMP | BPF_JA:
 		case BPF_JMP32 | BPF_JA:
@@ -2729,8 +2735,6 @@ st:			if (is_imm8(insn->off))
 			ctx->cleanup_addr = proglen;
 			if (bpf_prog_was_classic(bpf_prog) &&
 			    !ns_capable_noaudit(&init_user_ns, CAP_SYS_ADMIN)) {
-				u8 *ip = image + addrs[i - 1];
-
 				if (emit_spectre_bhb_barrier(&prog, ip, bpf_prog))
 					return -EINVAL;
 			}
@@ -3791,7 +3795,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
 	for (pass = 0; pass < MAX_PASSES || image; pass++) {
 		if (!padding && pass >= PADDING_PASSES)
 			padding = true;
-		proglen = do_jit(prog, addrs, image, rw_image, oldproglen, &ctx, padding);
+		proglen = do_jit(env, prog, addrs, image, rw_image, oldproglen, &ctx, padding);
 		if (proglen <= 0) {
 out_image:
 			image = NULL;
-- 
2.47.3



^ permalink raw reply related	[flat|nested] 20+ messages in thread

* Re: [bpf-next v8 4/5] bpf, x86: Emit ENDBR for indirect jump targets
  2026-03-09 14:00 ` [bpf-next v8 4/5] bpf, x86: Emit ENDBR for " Xu Kuohai
@ 2026-03-09 16:37   ` Anton Protopopov
  0 siblings, 0 replies; 20+ messages in thread
From: Anton Protopopov @ 2026-03-09 16:37 UTC (permalink / raw)
  To: Xu Kuohai
  Cc: bpf, linux-kernel, linux-arm-kernel, Alexei Starovoitov,
	Daniel Borkmann, Andrii Nakryiko, Martin KaFai Lau,
	Eduard Zingerman, Yonghong Song, Puranjay Mohan, Shahab Vahedi,
	Russell King, Tiezhu Yang, Hengqi Chen, Johan Almbladh,
	Paul Burton, Hari Bathini, Christophe Leroy, Naveen N Rao,
	Luke Nelson, Xi Wang, Björn Töpel, Pu Lehui,
	Ilya Leoshkevich, Heiko Carstens, Vasily Gorbik, David S . Miller,
	Wang YanQing

On 26/03/09 10:00PM, Xu Kuohai wrote:
> From: Xu Kuohai <xukuohai@huawei.com>
> 
> On CPUs that support CET/IBT, the indirect jump selftest triggers
> a kernel panic because the indirect jump targets lack ENDBR
> instructions.
> 
> To fix it, emit an ENDBR instruction to each indirect jump target. Since
> the ENDBR instruction shifts the position of original jited instructions,
> fix the instruction address calculation wherever the addresses are used.
> 
> For reference, below is a sample panic log.
> 
>  Missing ENDBR: bpf_prog_2e5f1c71c13ac3e0_big_jump_table+0x97/0xe1
>  ------------[ cut here ]------------
>  kernel BUG at arch/x86/kernel/cet.c:133!
>  Oops: invalid opcode: 0000 [#1] SMP NOPTI
> 
>  ...
> 
>   ? 0xffffffffc00fb258
>   ? bpf_prog_2e5f1c71c13ac3e0_big_jump_table+0x97/0xe1
>   bpf_prog_test_run_syscall+0x110/0x2f0
>   ? fdget+0xba/0xe0
>   __sys_bpf+0xe4b/0x2590
>   ? __kmalloc_node_track_caller_noprof+0x1c7/0x680
>   ? bpf_prog_test_run_syscall+0x215/0x2f0
>   __x64_sys_bpf+0x21/0x30
>   do_syscall_64+0x85/0x620
>   ? bpf_prog_test_run_syscall+0x1e2/0x2f0
> 
> Fixes: 493d9e0d6083 ("bpf, x86: add support for indirect jumps")
> Signed-off-by: Xu Kuohai <xukuohai@huawei.com>
> ---
>  arch/x86/net/bpf_jit_comp.c | 26 +++++++++++++++-----------
>  1 file changed, 15 insertions(+), 11 deletions(-)
> 
> diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
> index b95f23ad1093..251dff1cd8e4 100644
> --- a/arch/x86/net/bpf_jit_comp.c
> +++ b/arch/x86/net/bpf_jit_comp.c
> @@ -1649,8 +1649,8 @@ static int emit_spectre_bhb_barrier(u8 **pprog, u8 *ip,
>  	return 0;
>  }
>  
> -static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image,
> -		  int oldproglen, struct jit_context *ctx, bool jmp_padding)
> +static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *addrs, u8 *image,
> +		  u8 *rw_image, int oldproglen, struct jit_context *ctx, bool jmp_padding)
>  {
>  	bool tail_call_reachable = bpf_prog->aux->tail_call_reachable;
>  	struct bpf_insn *insn = bpf_prog->insnsi;
> @@ -1663,7 +1663,7 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image
>  	void __percpu *priv_stack_ptr;
>  	int i, excnt = 0;
>  	int ilen, proglen = 0;
> -	u8 *prog = temp;
> +	u8 *ip, *prog = temp;
>  	u32 stack_depth;
>  	int err;
>  
> @@ -1734,6 +1734,13 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image
>  				dst_reg = X86_REG_R9;
>  		}
>  
> +#ifdef CONFIG_X86_KERNEL_IBT
> +		if (bpf_insn_is_indirect_target(env, bpf_prog, i - 1))
> +			EMIT_ENDBR();
> +#endif
> +
> +		ip = image + addrs[i - 1] + (prog - temp);
> +
>  		switch (insn->code) {
>  			/* ALU */
>  		case BPF_ALU | BPF_ADD | BPF_X:
> @@ -2440,8 +2447,6 @@ st:			if (is_imm8(insn->off))
>  
>  			/* call */
>  		case BPF_JMP | BPF_CALL: {
> -			u8 *ip = image + addrs[i - 1];
> -
>  			func = (u8 *) __bpf_call_base + imm32;
>  			if (src_reg == BPF_PSEUDO_CALL && tail_call_reachable) {
>  				LOAD_TAIL_CALL_CNT_PTR(stack_depth);
> @@ -2465,7 +2470,8 @@ st:			if (is_imm8(insn->off))
>  			if (imm32)
>  				emit_bpf_tail_call_direct(bpf_prog,
>  							  &bpf_prog->aux->poke_tab[imm32 - 1],
> -							  &prog, image + addrs[i - 1],
> +							  &prog,
> +							  ip,
>  							  callee_regs_used,
>  							  stack_depth,
>  							  ctx);
> @@ -2474,7 +2480,7 @@ st:			if (is_imm8(insn->off))
>  							    &prog,
>  							    callee_regs_used,
>  							    stack_depth,
> -							    image + addrs[i - 1],
> +							    ip,
>  							    ctx);
>  			break;
>  
> @@ -2639,7 +2645,7 @@ st:			if (is_imm8(insn->off))
>  			break;
>  
>  		case BPF_JMP | BPF_JA | BPF_X:
> -			emit_indirect_jump(&prog, insn->dst_reg, image + addrs[i - 1]);
> +			emit_indirect_jump(&prog, insn->dst_reg, ip);
>  			break;
>  		case BPF_JMP | BPF_JA:
>  		case BPF_JMP32 | BPF_JA:
> @@ -2729,8 +2735,6 @@ st:			if (is_imm8(insn->off))
>  			ctx->cleanup_addr = proglen;
>  			if (bpf_prog_was_classic(bpf_prog) &&
>  			    !ns_capable_noaudit(&init_user_ns, CAP_SYS_ADMIN)) {
> -				u8 *ip = image + addrs[i - 1];
> -
>  				if (emit_spectre_bhb_barrier(&prog, ip, bpf_prog))
>  					return -EINVAL;
>  			}
> @@ -3791,7 +3795,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
>  	for (pass = 0; pass < MAX_PASSES || image; pass++) {
>  		if (!padding && pass >= PADDING_PASSES)
>  			padding = true;
> -		proglen = do_jit(prog, addrs, image, rw_image, oldproglen, &ctx, padding);
> +		proglen = do_jit(env, prog, addrs, image, rw_image, oldproglen, &ctx, padding);
>  		if (proglen <= 0) {
>  out_image:
>  			image = NULL;
> -- 
> 2.47.3

Reviewed-by: Anton Protopopov <a.s.protopopov@gmail.com>


^ permalink raw reply	[flat|nested] 20+ messages in thread

* [bpf-next v8 5/5] bpf, arm64: Emit BTI for indirect jump target
  2026-03-09 14:00 [bpf-next v8 0/5] emit ENDBR/BTI instructions for indirect jump targets Xu Kuohai
                   ` (3 preceding siblings ...)
  2026-03-09 14:00 ` [bpf-next v8 4/5] bpf, x86: Emit ENDBR for " Xu Kuohai
@ 2026-03-09 14:00 ` Xu Kuohai
  2026-03-09 16:38   ` Anton Protopopov
  2026-03-09 15:00 ` [bpf-next v8 0/5] emit ENDBR/BTI instructions for indirect jump targets Alexis Lothoré
  2026-03-09 17:34 ` Anton Protopopov
  6 siblings, 1 reply; 20+ messages in thread
From: Xu Kuohai @ 2026-03-09 14:00 UTC (permalink / raw)
  To: bpf, linux-kernel, linux-arm-kernel
  Cc: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
	Martin KaFai Lau, Eduard Zingerman, Yonghong Song, Puranjay Mohan,
	Anton Protopopov, Shahab Vahedi, Russell King, Tiezhu Yang,
	Hengqi Chen, Johan Almbladh, Paul Burton, Hari Bathini,
	Christophe Leroy, Naveen N Rao, Luke Nelson, Xi Wang,
	Björn Töpel, Pu Lehui, Ilya Leoshkevich, Heiko Carstens,
	Vasily Gorbik, David S . Miller, Wang YanQing

From: Xu Kuohai <xukuohai@huawei.com>

On CPUs that support BTI, the indirect jump selftest triggers a kernel
panic because there is no BTI instructions at the indirect jump targets.

Fix it by emitting a BTI instruction for each indirect jump target.

For reference, below is a sample panic log.

Internal error: Oops - BTI: 0000000036000003 [#1]  SMP
...
Call trace:
 bpf_prog_2e5f1c71c13ac3e0_big_jump_table+0x54/0xf8 (P)
 bpf_prog_run_pin_on_cpu+0x140/0x468
 bpf_prog_test_run_syscall+0x280/0x3b8
 bpf_prog_test_run+0x22c/0x2c0

Fixes: f4a66cf1cb14 ("bpf: arm64: Add support for indirect jumps")
Signed-off-by: Xu Kuohai <xukuohai@huawei.com>
---
 arch/arm64/net/bpf_jit_comp.c | 17 ++++++++++-------
 1 file changed, 10 insertions(+), 7 deletions(-)

diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
index 5c785eab4b5a..036ad607720a 100644
--- a/arch/arm64/net/bpf_jit_comp.c
+++ b/arch/arm64/net/bpf_jit_comp.c
@@ -1198,8 +1198,8 @@ static int add_exception_handler(const struct bpf_insn *insn,
  * >0 - successfully JITed a 16-byte eBPF instruction.
  * <0 - failed to JIT.
  */
-static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx,
-		      bool extra_pass)
+static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn *insn,
+		      struct jit_ctx *ctx, bool extra_pass)
 {
 	const u8 code = insn->code;
 	u8 dst = bpf2a64[insn->dst_reg];
@@ -1224,6 +1224,9 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx,
 	int ret;
 	bool sign_extend;
 
+	if (bpf_insn_is_indirect_target(env, ctx->prog, i))
+		emit_bti(A64_BTI_J, ctx);
+
 	switch (code) {
 	/* dst = src */
 	case BPF_ALU | BPF_MOV | BPF_X:
@@ -1899,7 +1902,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx,
 	return 0;
 }
 
-static int build_body(struct jit_ctx *ctx, bool extra_pass)
+static int build_body(struct bpf_verifier_env *env, struct jit_ctx *ctx, bool extra_pass)
 {
 	const struct bpf_prog *prog = ctx->prog;
 	int i;
@@ -1918,7 +1921,7 @@ static int build_body(struct jit_ctx *ctx, bool extra_pass)
 		int ret;
 
 		ctx->offset[i] = ctx->idx;
-		ret = build_insn(insn, ctx, extra_pass);
+		ret = build_insn(env, insn, ctx, extra_pass);
 		if (ret > 0) {
 			i++;
 			ctx->offset[i] = ctx->idx;
@@ -2086,7 +2089,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
 	if (build_prologue(&ctx, was_classic))
 		goto out_off;
 
-	if (build_body(&ctx, extra_pass))
+	if (build_body(env, &ctx, extra_pass))
 		goto out_off;
 
 	ctx.epilogue_offset = ctx.idx;
@@ -2134,7 +2137,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
 	/* Dont write body instructions to memory for now */
 	ctx.write = false;
 
-	if (build_body(&ctx, extra_pass))
+	if (build_body(env, &ctx, extra_pass))
 		goto out_free_hdr;
 
 	ctx.epilogue_offset = ctx.idx;
@@ -2143,7 +2146,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
 	ctx.write = true;
 
 	/* Pass 3: Adjust jump offset and write final image */
-	if (build_body(&ctx, extra_pass) ||
+	if (build_body(env, &ctx, extra_pass) ||
 		WARN_ON_ONCE(ctx.idx != ctx.epilogue_offset))
 		goto out_free_hdr;
 
-- 
2.47.3



^ permalink raw reply related	[flat|nested] 20+ messages in thread

* Re: [bpf-next v8 5/5] bpf, arm64: Emit BTI for indirect jump target
  2026-03-09 14:00 ` [bpf-next v8 5/5] bpf, arm64: Emit BTI for indirect jump target Xu Kuohai
@ 2026-03-09 16:38   ` Anton Protopopov
  0 siblings, 0 replies; 20+ messages in thread
From: Anton Protopopov @ 2026-03-09 16:38 UTC (permalink / raw)
  To: Xu Kuohai
  Cc: bpf, linux-kernel, linux-arm-kernel, Alexei Starovoitov,
	Daniel Borkmann, Andrii Nakryiko, Martin KaFai Lau,
	Eduard Zingerman, Yonghong Song, Puranjay Mohan, Shahab Vahedi,
	Russell King, Tiezhu Yang, Hengqi Chen, Johan Almbladh,
	Paul Burton, Hari Bathini, Christophe Leroy, Naveen N Rao,
	Luke Nelson, Xi Wang, Björn Töpel, Pu Lehui,
	Ilya Leoshkevich, Heiko Carstens, Vasily Gorbik, David S . Miller,
	Wang YanQing

On 26/03/09 10:00PM, Xu Kuohai wrote:
> From: Xu Kuohai <xukuohai@huawei.com>
> 
> On CPUs that support BTI, the indirect jump selftest triggers a kernel
> panic because there is no BTI instructions at the indirect jump targets.
> 
> Fix it by emitting a BTI instruction for each indirect jump target.
> 
> For reference, below is a sample panic log.
> 
> Internal error: Oops - BTI: 0000000036000003 [#1]  SMP
> ...
> Call trace:
>  bpf_prog_2e5f1c71c13ac3e0_big_jump_table+0x54/0xf8 (P)
>  bpf_prog_run_pin_on_cpu+0x140/0x468
>  bpf_prog_test_run_syscall+0x280/0x3b8
>  bpf_prog_test_run+0x22c/0x2c0
> 
> Fixes: f4a66cf1cb14 ("bpf: arm64: Add support for indirect jumps")
> Signed-off-by: Xu Kuohai <xukuohai@huawei.com>
> ---
>  arch/arm64/net/bpf_jit_comp.c | 17 ++++++++++-------
>  1 file changed, 10 insertions(+), 7 deletions(-)
> 
> diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
> index 5c785eab4b5a..036ad607720a 100644
> --- a/arch/arm64/net/bpf_jit_comp.c
> +++ b/arch/arm64/net/bpf_jit_comp.c
> @@ -1198,8 +1198,8 @@ static int add_exception_handler(const struct bpf_insn *insn,
>   * >0 - successfully JITed a 16-byte eBPF instruction.
>   * <0 - failed to JIT.
>   */
> -static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx,
> -		      bool extra_pass)
> +static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn *insn,
> +		      struct jit_ctx *ctx, bool extra_pass)
>  {
>  	const u8 code = insn->code;
>  	u8 dst = bpf2a64[insn->dst_reg];
> @@ -1224,6 +1224,9 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx,
>  	int ret;
>  	bool sign_extend;
>  
> +	if (bpf_insn_is_indirect_target(env, ctx->prog, i))
> +		emit_bti(A64_BTI_J, ctx);
> +
>  	switch (code) {
>  	/* dst = src */
>  	case BPF_ALU | BPF_MOV | BPF_X:
> @@ -1899,7 +1902,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx,
>  	return 0;
>  }
>  
> -static int build_body(struct jit_ctx *ctx, bool extra_pass)
> +static int build_body(struct bpf_verifier_env *env, struct jit_ctx *ctx, bool extra_pass)
>  {
>  	const struct bpf_prog *prog = ctx->prog;
>  	int i;
> @@ -1918,7 +1921,7 @@ static int build_body(struct jit_ctx *ctx, bool extra_pass)
>  		int ret;
>  
>  		ctx->offset[i] = ctx->idx;
> -		ret = build_insn(insn, ctx, extra_pass);
> +		ret = build_insn(env, insn, ctx, extra_pass);
>  		if (ret > 0) {
>  			i++;
>  			ctx->offset[i] = ctx->idx;
> @@ -2086,7 +2089,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
>  	if (build_prologue(&ctx, was_classic))
>  		goto out_off;
>  
> -	if (build_body(&ctx, extra_pass))
> +	if (build_body(env, &ctx, extra_pass))
>  		goto out_off;
>  
>  	ctx.epilogue_offset = ctx.idx;
> @@ -2134,7 +2137,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
>  	/* Dont write body instructions to memory for now */
>  	ctx.write = false;
>  
> -	if (build_body(&ctx, extra_pass))
> +	if (build_body(env, &ctx, extra_pass))
>  		goto out_free_hdr;
>  
>  	ctx.epilogue_offset = ctx.idx;
> @@ -2143,7 +2146,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
>  	ctx.write = true;
>  
>  	/* Pass 3: Adjust jump offset and write final image */
> -	if (build_body(&ctx, extra_pass) ||
> +	if (build_body(env, &ctx, extra_pass) ||
>  		WARN_ON_ONCE(ctx.idx != ctx.epilogue_offset))
>  		goto out_free_hdr;
>  
> -- 
> 2.47.3
> 

Reviewed-by: Anton Protopopov <a.s.protopopov@gmail.com>


^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [bpf-next v8 0/5] emit ENDBR/BTI instructions for indirect jump targets
  2026-03-09 14:00 [bpf-next v8 0/5] emit ENDBR/BTI instructions for indirect jump targets Xu Kuohai
                   ` (4 preceding siblings ...)
  2026-03-09 14:00 ` [bpf-next v8 5/5] bpf, arm64: Emit BTI for indirect jump target Xu Kuohai
@ 2026-03-09 15:00 ` Alexis Lothoré
  2026-03-10  6:25   ` Xu Kuohai
  2026-03-09 17:34 ` Anton Protopopov
  6 siblings, 1 reply; 20+ messages in thread
From: Alexis Lothoré @ 2026-03-09 15:00 UTC (permalink / raw)
  To: Xu Kuohai, bpf, linux-kernel, linux-arm-kernel
  Cc: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
	Martin KaFai Lau, Eduard Zingerman, Yonghong Song, Puranjay Mohan,
	Anton Protopopov, Shahab Vahedi, Russell King, Tiezhu Yang,
	Hengqi Chen, Johan Almbladh, Paul Burton, Hari Bathini,
	Christophe Leroy, Naveen N Rao, Luke Nelson, Xi Wang,
	Björn Töpel, Pu Lehui, Ilya Leoshkevich, Heiko Carstens,
	Vasily Gorbik, David S . Miller, Wang YanQing

Hi Xu,

On Mon Mar 9, 2026 at 3:00 PM CET, Xu Kuohai wrote:
> On architectures with CFI protection enabled that require landing pad
> instructions at indirect jump targets, such as x86 with CET/IBT eanbled
> and arm64 with BTI enabled, kernel panics when an indirect jump lands on
> a target witout landing pad. Therefore, the JIT must emit landing pad
> instructions for indirect jump targets.
>
> The verifier already recognizes which instructions are indirect jump
> targets during the verification phase. So we can stores this information
> in env->insn_aux_data and pass it to the JIT as new parameter, so the JIT
> knows which instructions are indirect jump targets.
>
> During JIT, constants blinding is performed. It rewrites the private copy
> of instructions for the JITed program, but it does not adjust the global
> env->insn_aux_data array. As a result, after constants blinding, the
> instruction indexes used by JIT may no longer match the indexes in
> env->insn_aux_data, so the JIT can not lookup env->insn_aux_data directly.
>
> To avoid this mistach, and considering that all existing arch-specific JITs
> already implement constants blinding with largely duplicated code, move
> constants blinding from JIT to generic code, before copying instructions
> for each subprog.

Could you please add me in CC for any future revision ?

Thanks,

Alexis

-- 
Alexis Lothoré, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com



^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [bpf-next v8 0/5] emit ENDBR/BTI instructions for indirect jump targets
  2026-03-09 15:00 ` [bpf-next v8 0/5] emit ENDBR/BTI instructions for indirect jump targets Alexis Lothoré
@ 2026-03-10  6:25   ` Xu Kuohai
  0 siblings, 0 replies; 20+ messages in thread
From: Xu Kuohai @ 2026-03-10  6:25 UTC (permalink / raw)
  To: Alexis Lothoré, bpf, linux-kernel, linux-arm-kernel
  Cc: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
	Martin KaFai Lau, Eduard Zingerman, Yonghong Song, Puranjay Mohan,
	Anton Protopopov, Shahab Vahedi, Russell King, Tiezhu Yang,
	Hengqi Chen, Johan Almbladh, Paul Burton, Hari Bathini,
	Christophe Leroy, Naveen N Rao, Luke Nelson, Xi Wang,
	Björn Töpel, Pu Lehui, Ilya Leoshkevich, Heiko Carstens,
	Vasily Gorbik, David S . Miller, Wang YanQing

On 3/9/2026 11:00 PM, Alexis Lothoré wrote:
> Hi Xu,
> 
> On Mon Mar 9, 2026 at 3:00 PM CET, Xu Kuohai wrote:
>> On architectures with CFI protection enabled that require landing pad
>> instructions at indirect jump targets, such as x86 with CET/IBT eanbled
>> and arm64 with BTI enabled, kernel panics when an indirect jump lands on
>> a target witout landing pad. Therefore, the JIT must emit landing pad
>> instructions for indirect jump targets.
>>
>> The verifier already recognizes which instructions are indirect jump
>> targets during the verification phase. So we can stores this information
>> in env->insn_aux_data and pass it to the JIT as new parameter, so the JIT
>> knows which instructions are indirect jump targets.
>>
>> During JIT, constants blinding is performed. It rewrites the private copy
>> of instructions for the JITed program, but it does not adjust the global
>> env->insn_aux_data array. As a result, after constants blinding, the
>> instruction indexes used by JIT may no longer match the indexes in
>> env->insn_aux_data, so the JIT can not lookup env->insn_aux_data directly.
>>
>> To avoid this mistach, and considering that all existing arch-specific JITs
>> already implement constants blinding with largely duplicated code, move
>> constants blinding from JIT to generic code, before copying instructions
>> for each subprog.
> 
> Could you please add me in CC for any future revision ?
>

Sure, will do.

> Thanks,
> 
> Alexis
> 



^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [bpf-next v8 0/5] emit ENDBR/BTI instructions for indirect jump targets
  2026-03-09 14:00 [bpf-next v8 0/5] emit ENDBR/BTI instructions for indirect jump targets Xu Kuohai
                   ` (5 preceding siblings ...)
  2026-03-09 15:00 ` [bpf-next v8 0/5] emit ENDBR/BTI instructions for indirect jump targets Alexis Lothoré
@ 2026-03-09 17:34 ` Anton Protopopov
  2026-03-10  6:55   ` Xu Kuohai
  6 siblings, 1 reply; 20+ messages in thread
From: Anton Protopopov @ 2026-03-09 17:34 UTC (permalink / raw)
  To: Xu Kuohai
  Cc: bpf, linux-kernel, linux-arm-kernel, Alexei Starovoitov,
	Daniel Borkmann, Andrii Nakryiko, Martin KaFai Lau,
	Eduard Zingerman, Yonghong Song, Puranjay Mohan, Shahab Vahedi,
	Russell King, Tiezhu Yang, Hengqi Chen, Johan Almbladh,
	Paul Burton, Hari Bathini, Christophe Leroy, Naveen N Rao,
	Luke Nelson, Xi Wang, Björn Töpel, Pu Lehui,
	Ilya Leoshkevich, Heiko Carstens, Vasily Gorbik, David S . Miller,
	Wang YanQing

On 26/03/09 10:00PM, Xu Kuohai wrote:
> On architectures with CFI protection enabled that require landing pad
> instructions at indirect jump targets, such as x86 with CET/IBT eanbled
                                                                  ^ enabled
> and arm64 with BTI enabled, kernel panics when an indirect jump lands on
> a target witout landing pad. Therefore, the JIT must emit landing pad
           ^ without
> instructions for indirect jump targets.
> 
> The verifier already recognizes which instructions are indirect jump
> targets during the verification phase. So we can stores this information
                                                   ^ store
> in env->insn_aux_data and pass it to the JIT as new parameter, so the JIT
> knows which instructions are indirect jump targets.
> 
> During JIT, constants blinding is performed. It rewrites the private copy
> of instructions for the JITed program, but it does not adjust the global
> env->insn_aux_data array. As a result, after constants blinding, the
> instruction indexes used by JIT may no longer match the indexes in
> env->insn_aux_data, so the JIT can not lookup env->insn_aux_data directly.
> 
> To avoid this mistach, and considering that all existing arch-specific JITs
                ^ mismatch?
> already implement constants blinding with largely duplicated code, move
> constants blinding from JIT to generic code, before copying instructions
> for each subprog.
> 
> v8:
> - Define void bpf_jit_blind_constants() function when CONFIG_BPF_JIT is not set 
> - Move indirect_target fixup for insn patching from bpf_jit_blind_constants()
>   to adjust_insn_aux_data()
> 
> v7: https://lore.kernel.org/bpf/20260307103949.2340104-1-xukuohai@huaweicloud.com
> - Move constants blinding logic back to bpf/core.c
> - Compute ip address before switch statement in x86 JIT
> - Clear JIT state from error path on arm64 and loongarch 
> 
> v6: https://lore.kernel.org/bpf/20260306102329.2056216-1-xukuohai@huaweicloud.com/
> - Move constants blinding from JIT to verifier
> - Move call to bpf_prog_select_runtime from bpf_prog_load to verifier
> 
> v5: https://lore.kernel.org/bpf/20260302102726.1126019-1-xukuohai@huaweicloud.com/
> - Switch to pass env to JIT directly to get rid of coping private insn_aux_data for
>   each prog
> 
> v4: https://lore.kernel.org/all/20260114093914.2403982-1-xukuohai@huaweicloud.com/
> - Switch to the approach proposed by Eduard, using insn_aux_data to indentify indirect
>   jump targets, and emit ENDBR on x86
> 
> v3: https://lore.kernel.org/bpf/20251227081033.240336-1-xukuohai@huaweicloud.com/
> - Get rid of unnecessary enum definition (Yonghong Song, Anton Protopopov)
> 
> v2: https://lore.kernel.org/bpf/20251223085447.139301-1-xukuohai@huaweicloud.com/
> - Exclude instruction arrays not used for indirect jumps (Anton Protopopov)
> 
> v1: https://lore.kernel.org/bpf/20251127140318.3944249-1-xukuohai@huaweicloud.com/
> 
> Xu Kuohai (5):
>   bpf: Move constants blinding from JIT to verifier
>   bpf: Pass bpf_verifier_env to JIT
>   bpf: Add helper to detect indirect jump targets
>   bpf, x86: Emit ENDBR for indirect jump targets
>   bpf, arm64: Emit BTI for indirect jump target
> 
>  arch/arc/net/bpf_jit_core.c      |  37 +++-----
>  arch/arm/net/bpf_jit_32.c        |  43 ++--------
>  arch/arm64/net/bpf_jit_comp.c    |  86 +++++++------------
>  arch/loongarch/net/bpf_jit.c     |  58 ++++---------
>  arch/mips/net/bpf_jit_comp.c     |  22 +----
>  arch/parisc/net/bpf_jit_core.c   |  40 ++-------
>  arch/powerpc/net/bpf_jit_comp.c  |  47 +++-------
>  arch/riscv/net/bpf_jit_core.c    |  47 +++-------
>  arch/s390/net/bpf_jit_comp.c     |  43 ++--------
>  arch/sparc/net/bpf_jit_comp_64.c |  43 ++--------
>  arch/x86/net/bpf_jit_comp.c      |  68 +++++----------
>  arch/x86/net/bpf_jit_comp32.c    |  35 ++------
>  include/linux/bpf.h              |   2 +
>  include/linux/bpf_verifier.h     |   9 +-
>  include/linux/filter.h           |  15 +++-
>  kernel/bpf/core.c                | 142 +++++++++----------------------
>  kernel/bpf/syscall.c             |   4 -
>  kernel/bpf/verifier.c            |  45 +++++++---
>  18 files changed, 233 insertions(+), 553 deletions(-)
> 
> -- 
> 2.47.3
> 


^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [bpf-next v8 0/5] emit ENDBR/BTI instructions for indirect jump targets
  2026-03-09 17:34 ` Anton Protopopov
@ 2026-03-10  6:55   ` Xu Kuohai
  0 siblings, 0 replies; 20+ messages in thread
From: Xu Kuohai @ 2026-03-10  6:55 UTC (permalink / raw)
  To: Anton Protopopov
  Cc: bpf, linux-kernel, linux-arm-kernel, Alexei Starovoitov,
	Daniel Borkmann, Andrii Nakryiko, Martin KaFai Lau,
	Eduard Zingerman, Yonghong Song, Puranjay Mohan, Shahab Vahedi,
	Russell King, Tiezhu Yang, Hengqi Chen, Johan Almbladh,
	Paul Burton, Hari Bathini, Christophe Leroy, Naveen N Rao,
	Luke Nelson, Xi Wang, Björn Töpel, Pu Lehui,
	Ilya Leoshkevich, Heiko Carstens, Vasily Gorbik, David S . Miller,
	Wang YanQing

On 3/10/2026 1:34 AM, Anton Protopopov wrote:
> On 26/03/09 10:00PM, Xu Kuohai wrote:
>> On architectures with CFI protection enabled that require landing pad
>> instructions at indirect jump targets, such as x86 with CET/IBT eanbled
>                                                                    ^ enabled
>> and arm64 with BTI enabled, kernel panics when an indirect jump lands on
>> a target witout landing pad. Therefore, the JIT must emit landing pad
>             ^ without
>> instructions for indirect jump targets.
>>
>> The verifier already recognizes which instructions are indirect jump
>> targets during the verification phase. So we can stores this information
>                                                     ^ store
>> in env->insn_aux_data and pass it to the JIT as new parameter, so the JIT
>> knows which instructions are indirect jump targets.
>>
>> During JIT, constants blinding is performed. It rewrites the private copy
>> of instructions for the JITed program, but it does not adjust the global
>> env->insn_aux_data array. As a result, after constants blinding, the
>> instruction indexes used by JIT may no longer match the indexes in
>> env->insn_aux_data, so the JIT can not lookup env->insn_aux_data directly.
>>
>> To avoid this mistach, and considering that all existing arch-specific JITs
>                  ^ mismatch?

I'll fix them and run a spelling check before sending the next version, thanks.



^ permalink raw reply	[flat|nested] 20+ messages in thread

end of thread, other threads:[~2026-03-17 10:56 UTC | newest]

Thread overview: 20+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-03-09 14:00 [bpf-next v8 0/5] emit ENDBR/BTI instructions for indirect jump targets Xu Kuohai
2026-03-09 14:00 ` [bpf-next v8 1/5] bpf: Move constants blinding from JIT to verifier Xu Kuohai
2026-03-09 17:20   ` Anton Protopopov
2026-03-10  6:52     ` Xu Kuohai
2026-03-09 21:25   ` Eduard Zingerman
2026-03-10  7:39     ` Xu Kuohai
2026-03-17 10:55   ` kernel test robot
2026-03-09 14:00 ` [bpf-next v8 2/5] bpf: Pass bpf_verifier_env to JIT Xu Kuohai
2026-03-09 16:56   ` Anton Protopopov
2026-03-10  6:44     ` Xu Kuohai
2026-03-09 14:00 ` [bpf-next v8 3/5] bpf: Add helper to detect indirect jump targets Xu Kuohai
2026-03-09 17:30   ` Anton Protopopov
2026-03-09 14:00 ` [bpf-next v8 4/5] bpf, x86: Emit ENDBR for " Xu Kuohai
2026-03-09 16:37   ` Anton Protopopov
2026-03-09 14:00 ` [bpf-next v8 5/5] bpf, arm64: Emit BTI for indirect jump target Xu Kuohai
2026-03-09 16:38   ` Anton Protopopov
2026-03-09 15:00 ` [bpf-next v8 0/5] emit ENDBR/BTI instructions for indirect jump targets Alexis Lothoré
2026-03-10  6:25   ` Xu Kuohai
2026-03-09 17:34 ` Anton Protopopov
2026-03-10  6:55   ` Xu Kuohai

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox