[PATCH v6 bpf-next 00/17] BPF indirect jumps

[PATCH v6 bpf-next 00/17] BPF indirect jumps

[Anton Protopopov]

This patchset implements a new type of map, instruction set, and uses
it to build support for indirect branches in BPF (on x86). (The same
map will be later used to provide support for indirect calls and static
keys.) See [1], [2] for more context.

Short table of contents:

  * Patches 1-7 implement the new map of type
    BPF_MAP_TYPE_INSN_SET and corresponding selftests. This map can
    be used to track the "original -> xlated -> jitted mapping" for
    a given program. Patches 6,7 add support for "blinded" variant.

  * Patches 8-12 implement the support for indirect jumps

  * Patches 13-17 add support for LLVM-compiled programs
    containing indirect jumps and selftests.

Since the v3 the jump table support was merged to LLVM and now can be
enabled with -mcpu=v4. See [3] for the PR which added the support.
The selftests, however, don't use LLVM for now: the verifier_gotox
is implemented in asm, and bpf_gotox is actually turned off, as
CI only runs with LLVM 20, and the indirect jumps are supported
starting from 22.

See individual patches for more details on the implementation details.

v5 -> v6 (this series):

  * instruction arrays:
    * better document `struct bpf_insn_array_value` (Eduard)
    * remove a condition in `bpf_insn_array_adjust_after_remove` (Eduard)
    * make userspace see original, xlated, and jitted indexes (+original) (Eduard)

  * indirect jumps, kernel:
    * reject writes to the map
    * reject unaligned ops
    * add a check what `w` is not outside the program in check_config for `gotox` (Eduard)
    * do not introduce unneeded `bpf_find_containing_subprog_idx`
    * simplify error processing for `bpf_find_containing_subprog` (Eduard)
    * add `insn_state |= DISCOVERED` when it's discovered (Eduard)
    * support SUB operations on PTR_TO_INSN (Eduard)
    * make `gotox_tmp_buf` a bpf_iarray and use helper to relocate it (Eduard)
    * rename fields of `bpf_iarray` to more generic (Eduard)
    * re-implement `visit_gotox_insn` in a loop (Eduard)
    * some minor cleanups (Eduard)

  * libbpf:
    * `struct reloc_desc`: add a comment about `union` (Eduard)
    * rename parameters of (and one other place in code) `{create,add}_jt_map` to `sym_off` (Eduard)
    * `create_jt_map`: check that size/off are 8-byte aligned (Eduard)

  * Selftests:
    * instruction array selftests:
      * only run tests on x86_64
      * write a more generic function to test things to reduce code (Eduard)
      * errno wasn't used in checks, so don't reset it (Eduard)
      * print `i`, `xlated_off` and `map_out[i]` here (Eduard)
    * added `verifier_gotox` selftests which do not depend on LLVM:
    * disabled `bpf_gotox` tests by default

  * other changes:
    * remove an extra function in bpf disasm (Eduard)
    * some minor cleanups in the insn_successors patch (Eduard)
    * update documentation in `Documentation/bpf/linux-notes.html` about jumps, now it is supported :)

v3 -> v4 -> v5 (https://lore.kernel.org/bpf/20250930125111.1269861-1-a.s.protopopov@gmail.com/):

  * [v4 -> v5] rebased on top of the last bpf-next/master

  * instruction arrays:
    * add copyright (Alexei)
    * remove mutexes, add frozen back (Alexei)
    * setup 1:1 prog-map correspondence using atomic_xchg
    * do not copy/paste array_map_get_next_key, add a common helper (Alexei)
    * misc minor code cleanups (Alexei)

  * indirect jumps, kernel side:
    * remove jt_allocated, just check if insn is gotox (Eduard)
    * use copy_register_state instead of individual copies (Eduard)
    * in push_stack is_speculative should be inherited (Eduard)
    * a few cleanups for insn_successors, including omitting error path (Eduard)
    * check if reserved fields are used when considering `gotox` instruction (Eduard)
    * read size and alignment of read from insn_array should be 8 (Eduard)
    * put buffer for sorting in subfun info and realloc to grow as needed (Eduard)
    * properly do `jump_point` / `prune_point` from `push_gotox_edge` (Eduard)
    * use range_within to check states (Eduard)
    * some minor cleanups and fix commit message (Eduard)

  * indirect jumps, libbpf side:
    * close map_fd in some error paths in create_jt_map (Andrii)
    * maps for jump tables are actually not closed at all, fix this (Andrii)
    * rename map from `jt` to `.jumptables` (Andrii)
    * use `errstr` in an error message (Andrii)
    * rephrase error message to look more standard (Andrii)
    * misc other minor renames and cleanups (Andrii)

  * selftests:
    * add the frozen selftest back
    * add a selftest for two jumps loading same table

  * some other changes:
    * rebase and split insn_successor changes into separate patch
    * use PTR_ERR_OR_ZERO in the push stack patch (Eduard)
    * indirect jumps on x86: properly re-read *pprog (Eduard)

v2 -> v3 (https://lore.kernel.org/bpf/20250918093850.455051-1-a.s.protopopov@gmail.com/):

  * fix build failure when CONFIG_BPF_SYSCALL is not set (kbuild-bot)
  * reformat bpftool help messages (Quentin)

v1 -> v2 (https://lore.kernel.org/bpf/20250913193922.1910480-1-a.s.protopopov@gmail.com/):

  * push_stack changes:
    * sanitize_speculative_path should just return int (Eduard)
    * return code from sanitize_speculative_path, not EFAULT (Eduard)
    * when BPF_COMPLEXITY_LIMIT_JMP_SEQ is reached, return E2BIG (Eduard)

  * indirect jumps:
    * omit support for .imm=fd in gotox, as we're not using it for now (Eduard)
    * struct jt -> struct bpf_iarray (Eduard)
    * insn_successors: rewrite the interface to just return a pointer (Eduard)
    * remove min_index/max_index, use umin_value/umax_value instead (Alexei, Eduard)
    * move emit_indirect_jump args change to the previous patch (Eduard)
    * add a comment to map_mem_size() (Eduard)
    * use verifier_bug for some error cases in check_indirect_jump (Eduard)
    * clear_insn_aux_data: use start,len instead of start,end (Eduard)
    * make regs[insn->dst_reg].type = PTR_TO_INSN part of check_mem_access (Eduard)

  * constant blinding changes:
    * make subprog_start adjustment better readable (Eduard)
    * do not set subprog len, it is already set (Eduard)

  * libbpf:
    * remove check that relocations from .rodata are ok (Anton)
    * do not freeze the map, it is not necessary anymore (Anton)
    * rename the goto_x -> gotox everywhere (Anton)
    * use u64 when parsing LLVM jump tables (Eduard)
    * split patch in two due to spaces->tabs change (Eduard)
    * split bpftool changes to bpftool patch (Andrii)
    * make sym_size it a union with ext_idx (Andrii)
    * properly copy/free the jumptables_data section from elf (Andrii)
    * a few cosmetic changes around create_jt_map (Andrii)
    * fix some comments + rewrite patch description (Andrii)
    * inline bpf_prog__append_subprog_offsets (Andrii)
    * subprog_sec_offst -> subprog_sec_off (Andrii)
    * !strcmp -> strcmp() == 0 (Andrii)
    * make some function names more readable (Andrii)
    * allocate table of subfunc offsets via libbpf_reallocarray (Andrii)

  * selftests:
    * squash insn_array* tests together (Anton)

  * fixed build warnings (kernel test robot)

RFC -> v1 (https://lore.kernel.org/bpf/20250816180631.952085-1-a.s.protopopov@gmail.com/):

  * I've tried to address all the comments provided by Alexei and
    Eduard in RFC. Will try to list the most important of them below.
  * One big change: move from older LLVM version [5] to newer [4].
    Now LLVM generates jump tables as symbols in the new special
    section ".jumptables". Another part of this change is that
    libbpf now doesn't try to link map load and goto *rX, as
    1) this is absolutely not reliable 2) for some use cases this
    is impossible (namely, when more than one jump table can be used
    in the same gotox instruction).
  * Added insn_successors() support (Alexei, Eduard). This includes
    getting rid of the ugly bpf_insn_set_iter_xlated_offset()
    interface (Eduard).
  * Removed hack for the unreachable instruction, as new LLVM thank to
    Eduard doesn't generate it.
  * Set mem_size for direct map access properly instead of hacking.
    Remove off>0 check. (Alexei)
  * Do not allocate new memory for min_index/max_index (Alexei, Eduard)
  * Information required during check_cfg is now cached to be reused
    later (Alexei + general logic for supporting multiple JT per jump)
  * Properly compare registers in regsafe (Alexei, Eduard)
  * Remove support for JMP32 (Eduard)
  * Better checks in adjust_ptr_min_max_vals (Eduard)
  * More selftests were added (but still there's room for more) which
    directly use gotox (Alexei)
  * More checks and verbose messages added
  * "unique pointers" are no more in the map

Links:
  1. https://lpc.events/event/18/contributions/1941/
  2. https://lwn.net/Articles/1017439/
  3. https://github.com/llvm/llvm-project/pull/149715
  4. https://github.com/llvm/llvm-project/pull/149715#issuecomment-3274833753
  6. rfc: https://lore.kernel.org/bpf/20250615085943.3871208-1-a.s.protopopov@gmail.com/

Anton Protopopov (17):
  bpf: fix the return value of push_stack
  bpf: save the start of functions in bpf_prog_aux
  bpf: generalize and export map_get_next_key for arrays
  bpf, x86: add new map type: instructions array
  selftests/bpf: add selftests for new insn_array map
  bpf: support instructions arrays with constants blinding
  selftests/bpf: test instructions arrays with blinding
  bpf, x86: allow indirect jumps to r8...r15
  bpf: make bpf_insn_successors to return a pointer
  bpf, x86: add support for indirect jumps
  bpf: disasm: add support for BPF_JMP|BPF_JA|BPF_X
  bpf, docs: do not state that indirect jumps are not supported
  libbpf: fix formatting of bpf_object__append_subprog_code
  libbpf: support llvm-generated indirect jumps
  bpftool: Recognize insn_array map type
  selftests/bpf: add new verifier_gotox test
  selftests/bpf: add C-level selftests for indirect jumps

 Documentation/bpf/linux-notes.rst             |   8 -
 arch/x86/net/bpf_jit_comp.c                   |  39 +-
 include/linux/bpf.h                           |  44 ++
 include/linux/bpf_types.h                     |   1 +
 include/linux/bpf_verifier.h                  |  23 +-
 include/uapi/linux/bpf.h                      |  21 +
 kernel/bpf/Makefile                           |   2 +-
 kernel/bpf/arraymap.c                         |  19 +-
 kernel/bpf/bpf_insn_array.c                   | 303 ++++++++++
 kernel/bpf/core.c                             |  21 +
 kernel/bpf/disasm.c                           |   3 +
 kernel/bpf/liveness.c                         |  39 +-
 kernel/bpf/log.c                              |   1 +
 kernel/bpf/syscall.c                          |  22 +
 kernel/bpf/verifier.c                         | 556 +++++++++++++++---
 .../bpf/bpftool/Documentation/bpftool-map.rst |   3 +-
 tools/bpf/bpftool/map.c                       |   3 +-
 tools/include/uapi/linux/bpf.h                |  21 +
 tools/lib/bpf/libbpf.c                        | 282 ++++++++-
 tools/lib/bpf/libbpf_probes.c                 |   4 +
 tools/lib/bpf/linker.c                        |  10 +-
 tools/testing/selftests/bpf/Makefile          |   4 +-
 .../selftests/bpf/prog_tests/bpf_gotox.c      | 185 ++++++
 .../selftests/bpf/prog_tests/bpf_insn_array.c | 492 ++++++++++++++++
 .../selftests/bpf/prog_tests/verifier.c       |   2 +
 tools/testing/selftests/bpf/progs/bpf_gotox.c | 414 +++++++++++++
 .../selftests/bpf/progs/verifier_gotox.c      | 277 +++++++++
 27 files changed, 2667 insertions(+), 132 deletions(-)
 create mode 100644 kernel/bpf/bpf_insn_array.c
 create mode 100644 tools/testing/selftests/bpf/prog_tests/bpf_gotox.c
 create mode 100644 tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c
 create mode 100644 tools/testing/selftests/bpf/progs/bpf_gotox.c
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_gotox.c

-- 
2.34.1

[PATCH v6 bpf-next 01/17] bpf: fix the return value of push_stack

[Anton Protopopov]

In [1] Eduard mentioned that on push_stack failure verifier code
should return -ENOMEM instead of -EFAULT. After checking with the
other call sites I've found that code randomly returns either -ENOMEM
or -EFAULT. This patch unifies the return values for the push_stack
(and similar push_async_cb) functions such that error codes are
always assigned properly.

  [1] https://lore.kernel.org/bpf/20250615085943.3871208-1-a.s.protopopov@gmail.com

Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
---
 kernel/bpf/verifier.c | 80 +++++++++++++++++++++----------------------
 1 file changed, 40 insertions(+), 40 deletions(-)

diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 9b4f6920f79b..80c99ef4cac5 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -2109,7 +2109,7 @@ static struct bpf_verifier_state *push_stack(struct bpf_verifier_env *env,
 
 	elem = kzalloc(sizeof(struct bpf_verifier_stack_elem), GFP_KERNEL_ACCOUNT);
 	if (!elem)
-		return NULL;
+		return ERR_PTR(-ENOMEM);
 
 	elem->insn_idx = insn_idx;
 	elem->prev_insn_idx = prev_insn_idx;
@@ -2119,12 +2119,12 @@ static struct bpf_verifier_state *push_stack(struct bpf_verifier_env *env,
 	env->stack_size++;
 	err = copy_verifier_state(&elem->st, cur);
 	if (err)
-		return NULL;
+		return ERR_PTR(-ENOMEM);
 	elem->st.speculative |= speculative;
 	if (env->stack_size > BPF_COMPLEXITY_LIMIT_JMP_SEQ) {
 		verbose(env, "The sequence of %d jumps is too complex.\n",
 			env->stack_size);
-		return NULL;
+		return ERR_PTR(-E2BIG);
 	}
 	if (elem->st.parent) {
 		++elem->st.parent->branches;
@@ -2919,7 +2919,7 @@ static struct bpf_verifier_state *push_async_cb(struct bpf_verifier_env *env,
 
 	elem = kzalloc(sizeof(struct bpf_verifier_stack_elem), GFP_KERNEL_ACCOUNT);
 	if (!elem)
-		return NULL;
+		return ERR_PTR(-ENOMEM);
 
 	elem->insn_idx = insn_idx;
 	elem->prev_insn_idx = prev_insn_idx;
@@ -2931,7 +2931,7 @@ static struct bpf_verifier_state *push_async_cb(struct bpf_verifier_env *env,
 		verbose(env,
 			"The sequence of %d jumps is too complex for async cb.\n",
 			env->stack_size);
-		return NULL;
+		return ERR_PTR(-E2BIG);
 	}
 	/* Unlike push_stack() do not copy_verifier_state().
 	 * The caller state doesn't matter.
@@ -2942,7 +2942,7 @@ static struct bpf_verifier_state *push_async_cb(struct bpf_verifier_env *env,
 	elem->st.in_sleepable = is_sleepable;
 	frame = kzalloc(sizeof(*frame), GFP_KERNEL_ACCOUNT);
 	if (!frame)
-		return NULL;
+		return ERR_PTR(-ENOMEM);
 	init_func_state(env, frame,
 			BPF_MAIN_FUNC /* callsite */,
 			0 /* frameno within this callchain */,
@@ -9045,8 +9045,8 @@ static int process_iter_next_call(struct bpf_verifier_env *env, int insn_idx,
 		prev_st = find_prev_entry(env, cur_st->parent, insn_idx);
 		/* branch out active iter state */
 		queued_st = push_stack(env, insn_idx + 1, insn_idx, false);
-		if (!queued_st)
-			return -ENOMEM;
+		if (IS_ERR(queued_st))
+			return PTR_ERR(queued_st);
 
 		queued_iter = get_iter_from_state(queued_st, meta);
 		queued_iter->iter.state = BPF_ITER_STATE_ACTIVE;
@@ -10616,8 +10616,8 @@ static int push_callback_call(struct bpf_verifier_env *env, struct bpf_insn *ins
 		async_cb = push_async_cb(env, env->subprog_info[subprog].start,
 					 insn_idx, subprog,
 					 is_async_cb_sleepable(env, insn));
-		if (!async_cb)
-			return -EFAULT;
+		if (IS_ERR(async_cb))
+			return PTR_ERR(async_cb);
 		callee = async_cb->frame[0];
 		callee->async_entry_cnt = caller->async_entry_cnt + 1;
 
@@ -10633,8 +10633,8 @@ static int push_callback_call(struct bpf_verifier_env *env, struct bpf_insn *ins
 	 * proceed with next instruction within current frame.
 	 */
 	callback_state = push_stack(env, env->subprog_info[subprog].start, insn_idx, false);
-	if (!callback_state)
-		return -ENOMEM;
+	if (IS_ERR(callback_state))
+		return PTR_ERR(callback_state);
 
 	err = setup_func_entry(env, subprog, insn_idx, set_callee_state_cb,
 			       callback_state);
@@ -13859,9 +13859,9 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 		struct bpf_reg_state *regs;
 
 		branch = push_stack(env, env->insn_idx + 1, env->insn_idx, false);
-		if (!branch) {
+		if (IS_ERR(branch)) {
 			verbose(env, "failed to push state for failed lock acquisition\n");
-			return -ENOMEM;
+			return PTR_ERR(branch);
 		}
 
 		regs = branch->frame[branch->curframe]->regs;
@@ -14316,16 +14316,15 @@ struct bpf_sanitize_info {
 	bool mask_to_left;
 };
 
-static struct bpf_verifier_state *
-sanitize_speculative_path(struct bpf_verifier_env *env,
-			  const struct bpf_insn *insn,
-			  u32 next_idx, u32 curr_idx)
+static int sanitize_speculative_path(struct bpf_verifier_env *env,
+				     const struct bpf_insn *insn,
+				     u32 next_idx, u32 curr_idx)
 {
 	struct bpf_verifier_state *branch;
 	struct bpf_reg_state *regs;
 
 	branch = push_stack(env, next_idx, curr_idx, true);
-	if (branch && insn) {
+	if (!IS_ERR(branch) && insn) {
 		regs = branch->frame[branch->curframe]->regs;
 		if (BPF_SRC(insn->code) == BPF_K) {
 			mark_reg_unknown(env, regs, insn->dst_reg);
@@ -14334,7 +14333,7 @@ sanitize_speculative_path(struct bpf_verifier_env *env,
 			mark_reg_unknown(env, regs, insn->src_reg);
 		}
 	}
-	return branch;
+	return PTR_ERR_OR_ZERO(branch);
 }
 
 static int sanitize_ptr_alu(struct bpf_verifier_env *env,
@@ -14353,7 +14352,6 @@ static int sanitize_ptr_alu(struct bpf_verifier_env *env,
 	u8 opcode = BPF_OP(insn->code);
 	u32 alu_state, alu_limit;
 	struct bpf_reg_state tmp;
-	bool ret;
 	int err;
 
 	if (can_skip_alu_sanitation(env, insn))
@@ -14426,11 +14424,12 @@ static int sanitize_ptr_alu(struct bpf_verifier_env *env,
 		tmp = *dst_reg;
 		copy_register_state(dst_reg, ptr_reg);
 	}
-	ret = sanitize_speculative_path(env, NULL, env->insn_idx + 1,
-					env->insn_idx);
-	if (!ptr_is_dst_reg && ret)
+	err = sanitize_speculative_path(env, NULL, env->insn_idx + 1, env->insn_idx);
+	if (err < 0)
+		return REASON_STACK;
+	if (!ptr_is_dst_reg)
 		*dst_reg = tmp;
-	return !ret ? REASON_STACK : 0;
+	return 0;
 }
 
 static void sanitize_mark_insn_seen(struct bpf_verifier_env *env)
@@ -16750,8 +16749,8 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env,
 
 		/* branch out 'fallthrough' insn as a new state to explore */
 		queued_st = push_stack(env, idx + 1, idx, false);
-		if (!queued_st)
-			return -ENOMEM;
+		if (IS_ERR(queued_st))
+			return PTR_ERR(queued_st);
 
 		queued_st->may_goto_depth++;
 		if (prev_st)
@@ -16829,10 +16828,11 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env,
 		 * the fall-through branch for simulation under speculative
 		 * execution.
 		 */
-		if (!env->bypass_spec_v1 &&
-		    !sanitize_speculative_path(env, insn, *insn_idx + 1,
-					       *insn_idx))
-			return -EFAULT;
+		if (!env->bypass_spec_v1) {
+			err = sanitize_speculative_path(env, insn, *insn_idx + 1, *insn_idx);
+			if (err < 0)
+				return err;
+		}
 		if (env->log.level & BPF_LOG_LEVEL)
 			print_insn_state(env, this_branch, this_branch->curframe);
 		*insn_idx += insn->off;
@@ -16842,11 +16842,12 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env,
 		 * program will go. If needed, push the goto branch for
 		 * simulation under speculative execution.
 		 */
-		if (!env->bypass_spec_v1 &&
-		    !sanitize_speculative_path(env, insn,
-					       *insn_idx + insn->off + 1,
-					       *insn_idx))
-			return -EFAULT;
+		if (!env->bypass_spec_v1) {
+			err = sanitize_speculative_path(env, insn, *insn_idx + insn->off + 1,
+							*insn_idx);
+			if (err < 0)
+				return err;
+		}
 		if (env->log.level & BPF_LOG_LEVEL)
 			print_insn_state(env, this_branch, this_branch->curframe);
 		return 0;
@@ -16867,10 +16868,9 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env,
 			return err;
 	}
 
-	other_branch = push_stack(env, *insn_idx + insn->off + 1, *insn_idx,
-				  false);
-	if (!other_branch)
-		return -EFAULT;
+	other_branch = push_stack(env, *insn_idx + insn->off + 1, *insn_idx, false);
+	if (IS_ERR(other_branch))
+		return PTR_ERR(other_branch);
 	other_branch_regs = other_branch->frame[other_branch->curframe]->regs;
 
 	if (BPF_SRC(insn->code) == BPF_X) {
-- 
2.34.1

[PATCH v6 bpf-next 02/17] bpf: save the start of functions in bpf_prog_aux

[Anton Protopopov]

Introduce a new subprog_start field in bpf_prog_aux. This field may
be used by JIT compilers wanting to know the real absolute xlated
offset of the function being jitted. The func_info[func_id] may have
served this purpose, but func_info may be NULL, so JIT compilers
can't rely on it.

Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
---
 include/linux/bpf.h   | 1 +
 kernel/bpf/verifier.c | 1 +
 2 files changed, 2 insertions(+)

diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 204f9c759a41..3bda915cd7a8 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -1623,6 +1623,7 @@ struct bpf_prog_aux {
 	u32 ctx_arg_info_size;
 	u32 max_rdonly_access;
 	u32 max_rdwr_access;
+	u32 subprog_start;
 	struct btf *attach_btf;
 	struct bpf_ctx_arg_aux *ctx_arg_info;
 	void __percpu *priv_stack_ptr;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 80c99ef4cac5..4579082068ca 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -21607,6 +21607,7 @@ static int jit_subprogs(struct bpf_verifier_env *env)
 		func[i]->aux->func_idx = i;
 		/* Below members will be freed only at prog->aux */
 		func[i]->aux->btf = prog->aux->btf;
+		func[i]->aux->subprog_start = subprog_start;
 		func[i]->aux->func_info = prog->aux->func_info;
 		func[i]->aux->func_info_cnt = prog->aux->func_info_cnt;
 		func[i]->aux->poke_tab = prog->aux->poke_tab;
-- 
2.34.1

[PATCH v6 bpf-next 03/17] bpf: generalize and export map_get_next_key for arrays

[Anton Protopopov]

The kernel/bpf/array.c file defines the array_map_get_next_key()
function which finds the next key for array maps. It actually doesn't
use any map fields besides the generic max_entries field. Generalize
it, and export as bpf_array_get_next_key() such that it can be
re-used by other array-like maps.

Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
---
 include/linux/bpf.h   |  6 ++++++
 kernel/bpf/arraymap.c | 19 +++++++++----------
 2 files changed, 15 insertions(+), 10 deletions(-)

diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 3bda915cd7a8..e53cda0aabb6 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -2107,6 +2107,12 @@ struct bpf_array {
 	};
 };
 
+/*
+ * The bpf_array_get_next_key() function may be used for all array-like
+ * maps, i.e., maps with u32 keys with range [0 ,..., max_entries)
+ */
+int bpf_array_get_next_key(struct bpf_map *map, void *key, void *next_key);
+
 #define BPF_COMPLEXITY_LIMIT_INSNS      1000000 /* yes. 1M insns */
 #define MAX_TAIL_CALL_CNT 33
 
diff --git a/kernel/bpf/arraymap.c b/kernel/bpf/arraymap.c
index 0ba790c2d2e5..1eeb31c5b317 100644
--- a/kernel/bpf/arraymap.c
+++ b/kernel/bpf/arraymap.c
@@ -335,18 +335,17 @@ int bpf_percpu_array_copy(struct bpf_map *map, void *key, void *value)
 }
 
 /* Called from syscall */
-static int array_map_get_next_key(struct bpf_map *map, void *key, void *next_key)
+int bpf_array_get_next_key(struct bpf_map *map, void *key, void *next_key)
 {
-	struct bpf_array *array = container_of(map, struct bpf_array, map);
 	u32 index = key ? *(u32 *)key : U32_MAX;
 	u32 *next = (u32 *)next_key;
 
-	if (index >= array->map.max_entries) {
+	if (index >= map->max_entries) {
 		*next = 0;
 		return 0;
 	}
 
-	if (index == array->map.max_entries - 1)
+	if (index == map->max_entries - 1)
 		return -ENOENT;
 
 	*next = index + 1;
@@ -789,7 +788,7 @@ const struct bpf_map_ops array_map_ops = {
 	.map_alloc_check = array_map_alloc_check,
 	.map_alloc = array_map_alloc,
 	.map_free = array_map_free,
-	.map_get_next_key = array_map_get_next_key,
+	.map_get_next_key = bpf_array_get_next_key,
 	.map_release_uref = array_map_free_internal_structs,
 	.map_lookup_elem = array_map_lookup_elem,
 	.map_update_elem = array_map_update_elem,
@@ -815,7 +814,7 @@ const struct bpf_map_ops percpu_array_map_ops = {
 	.map_alloc_check = array_map_alloc_check,
 	.map_alloc = array_map_alloc,
 	.map_free = array_map_free,
-	.map_get_next_key = array_map_get_next_key,
+	.map_get_next_key = bpf_array_get_next_key,
 	.map_lookup_elem = percpu_array_map_lookup_elem,
 	.map_gen_lookup = percpu_array_map_gen_lookup,
 	.map_update_elem = array_map_update_elem,
@@ -1204,7 +1203,7 @@ const struct bpf_map_ops prog_array_map_ops = {
 	.map_poke_track = prog_array_map_poke_track,
 	.map_poke_untrack = prog_array_map_poke_untrack,
 	.map_poke_run = prog_array_map_poke_run,
-	.map_get_next_key = array_map_get_next_key,
+	.map_get_next_key = bpf_array_get_next_key,
 	.map_lookup_elem = fd_array_map_lookup_elem,
 	.map_delete_elem = fd_array_map_delete_elem,
 	.map_fd_get_ptr = prog_fd_array_get_ptr,
@@ -1308,7 +1307,7 @@ const struct bpf_map_ops perf_event_array_map_ops = {
 	.map_alloc_check = fd_array_map_alloc_check,
 	.map_alloc = array_map_alloc,
 	.map_free = perf_event_fd_array_map_free,
-	.map_get_next_key = array_map_get_next_key,
+	.map_get_next_key = bpf_array_get_next_key,
 	.map_lookup_elem = fd_array_map_lookup_elem,
 	.map_delete_elem = fd_array_map_delete_elem,
 	.map_fd_get_ptr = perf_event_fd_array_get_ptr,
@@ -1344,7 +1343,7 @@ const struct bpf_map_ops cgroup_array_map_ops = {
 	.map_alloc_check = fd_array_map_alloc_check,
 	.map_alloc = array_map_alloc,
 	.map_free = cgroup_fd_array_free,
-	.map_get_next_key = array_map_get_next_key,
+	.map_get_next_key = bpf_array_get_next_key,
 	.map_lookup_elem = fd_array_map_lookup_elem,
 	.map_delete_elem = fd_array_map_delete_elem,
 	.map_fd_get_ptr = cgroup_fd_array_get_ptr,
@@ -1429,7 +1428,7 @@ const struct bpf_map_ops array_of_maps_map_ops = {
 	.map_alloc_check = fd_array_map_alloc_check,
 	.map_alloc = array_of_map_alloc,
 	.map_free = array_of_map_free,
-	.map_get_next_key = array_map_get_next_key,
+	.map_get_next_key = bpf_array_get_next_key,
 	.map_lookup_elem = array_of_map_lookup_elem,
 	.map_delete_elem = fd_array_map_delete_elem,
 	.map_fd_get_ptr = bpf_map_fd_get_ptr,
-- 
2.34.1

[PATCH v6 bpf-next 04/17] bpf, x86: add new map type: instructions array

[Anton Protopopov]

On bpf(BPF_PROG_LOAD) syscall user-supplied BPF programs are
translated by the verifier into "xlated" BPF programs. During this
process the original instructions offsets might be adjusted and/or
individual instructions might be replaced by new sets of instructions,
or deleted.

Add a new BPF map type which is aimed to keep track of how, for a
given program, the original instructions were relocated during the
verification. Also, besides keeping track of the original -> xlated
mapping, make x86 JIT to build the xlated -> jitted mapping for every
instruction listed in an instruction array. This is required for every
future application of instruction arrays: static keys, indirect jumps
and indirect calls.

A map of the BPF_MAP_TYPE_INSN_ARRAY type must be created with a u32
keys and value of size 8. The values have different semantics for
userspace and for BPF space. For userspace a value consists of two
u32 values – xlated and jitted offsets. For BPF side the value is
a real pointer to a jitted instruction.

On map creation/initialization, before loading the program, each
element of the map should be initialized to point to an instruction
offset within the program. Before the program load such maps should
be made frozen. After the program verification xlated and jitted
offsets can be read via the bpf(2) syscall.

If a tracked instruction is removed by the verifier, then the xlated
offset is set to (u32)-1 which is considered to be too big for a valid
BPF program offset.

One such a map can, obviously, be used to track one and only one BPF
program.  If the verification process was unsuccessful, then the same
map can be re-used to verify the program with a different log level.
However, if the program was loaded fine, then such a map, being
frozen in any case, can't be reused by other programs even after the
program release.

Example. Consider the following original and xlated programs:

    Original prog:                      Xlated prog:

     0:  r1 = 0x0                        0: r1 = 0
     1:  *(u32 *)(r10 - 0x4) = r1        1: *(u32 *)(r10 -4) = r1
     2:  r2 = r10                        2: r2 = r10
     3:  r2 += -0x4                      3: r2 += -4
     4:  r1 = 0x0 ll                     4: r1 = map[id:88]
     6:  call 0x1                        6: r1 += 272
                                         7: r0 = *(u32 *)(r2 +0)
                                         8: if r0 >= 0x1 goto pc+3
                                         9: r0 <<= 3
                                        10: r0 += r1
                                        11: goto pc+1
                                        12: r0 = 0
     7:  r6 = r0                        13: r6 = r0
     8:  if r6 == 0x0 goto +0x2         14: if r6 == 0x0 goto pc+4
     9:  call 0x76                      15: r0 = 0xffffffff8d2079c0
                                        17: r0 = *(u64 *)(r0 +0)
    10:  *(u64 *)(r6 + 0x0) = r0        18: *(u64 *)(r6 +0) = r0
    11:  r0 = 0x0                       19: r0 = 0x0
    12:  exit                           20: exit

An instruction array map, containing, e.g., instructions [0,4,7,12]
will be translated by the verifier to [0,4,13,20]. A map with
index 5 (the middle of 16-byte instruction) or indexes greater than 12
(outside the program boundaries) would be rejected.

The functionality provided by this patch will be extended in consequent
patches to implement BPF Static Keys, indirect jumps, and indirect calls.

Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
---
 arch/x86/net/bpf_jit_comp.c    |   8 +
 include/linux/bpf.h            |  36 +++++
 include/linux/bpf_types.h      |   1 +
 include/linux/bpf_verifier.h   |   2 +
 include/uapi/linux/bpf.h       |  21 +++
 kernel/bpf/Makefile            |   2 +-
 kernel/bpf/bpf_insn_array.c    | 288 +++++++++++++++++++++++++++++++++
 kernel/bpf/syscall.c           |  22 +++
 kernel/bpf/verifier.c          |  43 +++++
 tools/include/uapi/linux/bpf.h |  21 +++
 10 files changed, 443 insertions(+), 1 deletion(-)
 create mode 100644 kernel/bpf/bpf_insn_array.c

diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index d4c93d9e73e4..c8e628410d2c 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -1691,6 +1691,7 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image
 	prog = temp;
 
 	for (i = 1; i <= insn_cnt; i++, insn++) {
+		u32 abs_xlated_off = bpf_prog->aux->subprog_start + i - 1;
 		const s32 imm32 = insn->imm;
 		u32 dst_reg = insn->dst_reg;
 		u32 src_reg = insn->src_reg;
@@ -2751,6 +2752,13 @@ st:			if (is_imm8(insn->off))
 				return -EFAULT;
 			}
 			memcpy(rw_image + proglen, temp, ilen);
+
+			/*
+			 * Instruction arrays need to know how xlated code
+			 * maps to jitted code
+			 */
+			bpf_prog_update_insn_ptr(bpf_prog, abs_xlated_off, proglen,
+						 image + proglen);
 		}
 		proglen += ilen;
 		addrs[i] = proglen;
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index e53cda0aabb6..363355628d2e 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -3789,4 +3789,40 @@ int bpf_prog_get_file_line(struct bpf_prog *prog, unsigned long ip, const char *
 			   const char **linep, int *nump);
 struct bpf_prog *bpf_prog_find_from_stack(void);
 
+int bpf_insn_array_init(struct bpf_map *map, const struct bpf_prog *prog);
+int bpf_insn_array_ready(struct bpf_map *map);
+void bpf_insn_array_release(struct bpf_map *map);
+void bpf_insn_array_adjust(struct bpf_map *map, u32 off, u32 len);
+void bpf_insn_array_adjust_after_remove(struct bpf_map *map, u32 off, u32 len);
+
+/*
+ * The struct bpf_insn_ptr structure describes a pointer to a
+ * particular instruction in a loaded BPF program. Initially
+ * it is initialised from userspace via user_value.xlated_off.
+ * During the program verification all other fields are populated
+ * accordingly:
+ *
+ *   jitted_ip:       address of the instruction in the jitted image
+ *   user_value:      user-visible original, xlated, and jitted offsets
+ */
+struct bpf_insn_ptr {
+	void *jitted_ip;
+	struct bpf_insn_array_value user_value;
+};
+
+#ifdef CONFIG_BPF_SYSCALL
+void bpf_prog_update_insn_ptr(struct bpf_prog *prog,
+			      u32 xlated_off,
+			      u32 jitted_off,
+			      void *jitted_ip);
+#else
+static inline void
+bpf_prog_update_insn_ptr(struct bpf_prog *prog,
+			 u32 xlated_off,
+			 u32 jitted_off,
+			 void *jitted_ip)
+{
+}
+#endif
+
 #endif /* _LINUX_BPF_H */
diff --git a/include/linux/bpf_types.h b/include/linux/bpf_types.h
index fa78f49d4a9a..b13de31e163f 100644
--- a/include/linux/bpf_types.h
+++ b/include/linux/bpf_types.h
@@ -133,6 +133,7 @@ BPF_MAP_TYPE(BPF_MAP_TYPE_RINGBUF, ringbuf_map_ops)
 BPF_MAP_TYPE(BPF_MAP_TYPE_BLOOM_FILTER, bloom_filter_map_ops)
 BPF_MAP_TYPE(BPF_MAP_TYPE_USER_RINGBUF, user_ringbuf_map_ops)
 BPF_MAP_TYPE(BPF_MAP_TYPE_ARENA, arena_map_ops)
+BPF_MAP_TYPE(BPF_MAP_TYPE_INSN_ARRAY, insn_array_map_ops)
 
 BPF_LINK_TYPE(BPF_LINK_TYPE_RAW_TRACEPOINT, raw_tracepoint)
 BPF_LINK_TYPE(BPF_LINK_TYPE_TRACING, tracing)
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index b57222a25a4a..142030e7e857 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -745,8 +745,10 @@ struct bpf_verifier_env {
 	struct list_head free_list;	/* list of struct bpf_verifier_state_list */
 	struct bpf_map *used_maps[MAX_USED_MAPS]; /* array of map's used by eBPF program */
 	struct btf_mod_pair used_btfs[MAX_USED_BTFS]; /* array of BTF's used by BPF program */
+	struct bpf_map *insn_array_maps[MAX_USED_MAPS]; /* array of INSN_ARRAY map's to be relocated */
 	u32 used_map_cnt;		/* number of used maps */
 	u32 used_btf_cnt;		/* number of used BTF objects */
+	u32 insn_array_map_cnt;		/* number of used maps of type BPF_MAP_TYPE_INSN_ARRAY */
 	u32 id_gen;			/* used to generate unique reg IDs */
 	u32 hidden_subprog_cnt;		/* number of hidden subprogs */
 	int exception_callback_subprog;
diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
index 6829936d33f5..805d441363cd 100644
--- a/include/uapi/linux/bpf.h
+++ b/include/uapi/linux/bpf.h
@@ -1026,6 +1026,7 @@ enum bpf_map_type {
 	BPF_MAP_TYPE_USER_RINGBUF,
 	BPF_MAP_TYPE_CGRP_STORAGE,
 	BPF_MAP_TYPE_ARENA,
+	BPF_MAP_TYPE_INSN_ARRAY,
 	__MAX_BPF_MAP_TYPE
 };
 
@@ -7645,4 +7646,24 @@ enum bpf_kfunc_flags {
 	BPF_F_PAD_ZEROS = (1ULL << 0),
 };
 
+/*
+ * Values of a BPF_MAP_TYPE_INSN_ARRAY entry must be of this type.
+ *
+ * Before the map is used the orig_off field should point to an
+ * instruction inside the program being loaded. The other fields
+ * must be set to 0.
+ *
+ * After the program is loaded, the xlated_off will be adjusted
+ * by the verifier to point to the index of the original instruction
+ * in the xlated program. If the instruction is deleted, it will
+ * be set to (u32)-1. The jitted_off will be set to the corresponding
+ * offset in the jitted image of the program.
+ */
+struct bpf_insn_array_value {
+	__u32 orig_off;
+	__u32 xlated_off;
+	__u32 jitted_off;
+	__u32 :32;
+};
+
 #endif /* _UAPI__LINUX_BPF_H__ */
diff --git a/kernel/bpf/Makefile b/kernel/bpf/Makefile
index 7fd0badfacb1..232cbc97434d 100644
--- a/kernel/bpf/Makefile
+++ b/kernel/bpf/Makefile
@@ -9,7 +9,7 @@ CFLAGS_core.o += -Wno-override-init $(cflags-nogcse-yy)
 obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o log.o token.o liveness.o
 obj-$(CONFIG_BPF_SYSCALL) += bpf_iter.o map_iter.o task_iter.o prog_iter.o link_iter.o
 obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o bloom_filter.o
-obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o
+obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o bpf_insn_array.o
 obj-$(CONFIG_BPF_SYSCALL) += bpf_local_storage.o bpf_task_storage.o
 obj-${CONFIG_BPF_LSM}	  += bpf_inode_storage.o
 obj-$(CONFIG_BPF_SYSCALL) += disasm.o mprog.o
diff --git a/kernel/bpf/bpf_insn_array.c b/kernel/bpf/bpf_insn_array.c
new file mode 100644
index 000000000000..fa2c162a7264
--- /dev/null
+++ b/kernel/bpf/bpf_insn_array.c
@@ -0,0 +1,288 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright (c) 2025 Isovalent */
+
+#include <linux/bpf.h>
+
+#define MAX_INSN_ARRAY_ENTRIES 256
+
+struct bpf_insn_array {
+	struct bpf_map map;
+	atomic_t used;
+	long *ips;
+	DECLARE_FLEX_ARRAY(struct bpf_insn_ptr, ptrs);
+};
+
+#define cast_insn_array(MAP_PTR) \
+	container_of((MAP_PTR), struct bpf_insn_array, map)
+
+#define INSN_DELETED ((u32)-1)
+
+static inline u32 insn_array_alloc_size(u32 max_entries)
+{
+	const u32 base_size = sizeof(struct bpf_insn_array);
+	const u32 entry_size = sizeof(struct bpf_insn_ptr);
+
+	return base_size + entry_size * max_entries;
+}
+
+static int insn_array_alloc_check(union bpf_attr *attr)
+{
+	u32 value_size = sizeof(struct bpf_insn_array_value);
+
+	if (attr->max_entries == 0 || attr->key_size != 4 ||
+	    attr->value_size != value_size || attr->map_flags != 0)
+		return -EINVAL;
+
+	if (attr->max_entries > MAX_INSN_ARRAY_ENTRIES)
+		return -E2BIG;
+
+	return 0;
+}
+
+static void insn_array_free(struct bpf_map *map)
+{
+	struct bpf_insn_array *insn_array = cast_insn_array(map);
+
+	kfree(insn_array->ips);
+	bpf_map_area_free(insn_array);
+}
+
+static struct bpf_map *insn_array_alloc(union bpf_attr *attr)
+{
+	u64 size = insn_array_alloc_size(attr->max_entries);
+	struct bpf_insn_array *insn_array;
+
+	insn_array = bpf_map_area_alloc(size, NUMA_NO_NODE);
+	if (!insn_array)
+		return ERR_PTR(-ENOMEM);
+
+	insn_array->ips = kcalloc(attr->max_entries, sizeof(long), GFP_KERNEL);
+	if (!insn_array->ips) {
+		insn_array_free(&insn_array->map);
+		return ERR_PTR(-ENOMEM);
+	}
+
+	bpf_map_init_from_attr(&insn_array->map, attr);
+
+	return &insn_array->map;
+}
+
+static void *insn_array_lookup_elem(struct bpf_map *map, void *key)
+{
+	struct bpf_insn_array *insn_array = cast_insn_array(map);
+	u32 index = *(u32 *)key;
+
+	if (unlikely(index >= insn_array->map.max_entries))
+		return NULL;
+
+	return &insn_array->ptrs[index].user_value;
+}
+
+static long insn_array_update_elem(struct bpf_map *map, void *key, void *value, u64 map_flags)
+{
+	struct bpf_insn_array *insn_array = cast_insn_array(map);
+	u32 index = *(u32 *)key;
+	struct bpf_insn_array_value val = {};
+
+	if (unlikely(index >= insn_array->map.max_entries))
+		return -E2BIG;
+
+	if (unlikely(map_flags & BPF_NOEXIST))
+		return -EEXIST;
+
+	copy_map_value(map, &val, value);
+	if (val.jitted_off || val.xlated_off)
+		return -EINVAL;
+
+	insn_array->ptrs[index].user_value.orig_off = val.orig_off;
+
+	return 0;
+}
+
+static long insn_array_delete_elem(struct bpf_map *map, void *key)
+{
+	return -EINVAL;
+}
+
+static int insn_array_check_btf(const struct bpf_map *map,
+			      const struct btf *btf,
+			      const struct btf_type *key_type,
+			      const struct btf_type *value_type)
+{
+	if (!btf_type_is_i32(key_type))
+		return -EINVAL;
+
+	if (!btf_type_is_i64(value_type))
+		return -EINVAL;
+
+	return 0;
+}
+
+static u64 insn_array_mem_usage(const struct bpf_map *map)
+{
+	u64 extra_size = 0;
+
+	extra_size += sizeof(long) * map->max_entries; /* insn_array->ips */
+
+	return insn_array_alloc_size(map->max_entries) + extra_size;
+}
+
+BTF_ID_LIST_SINGLE(insn_array_btf_ids, struct, bpf_insn_array)
+
+const struct bpf_map_ops insn_array_map_ops = {
+	.map_alloc_check = insn_array_alloc_check,
+	.map_alloc = insn_array_alloc,
+	.map_free = insn_array_free,
+	.map_get_next_key = bpf_array_get_next_key,
+	.map_lookup_elem = insn_array_lookup_elem,
+	.map_update_elem = insn_array_update_elem,
+	.map_delete_elem = insn_array_delete_elem,
+	.map_check_btf = insn_array_check_btf,
+	.map_mem_usage = insn_array_mem_usage,
+	.map_btf_id = &insn_array_btf_ids[0],
+};
+
+static inline bool is_frozen(struct bpf_map *map)
+{
+	guard(mutex)(&map->freeze_mutex);
+
+	return map->frozen;
+}
+
+static bool is_insn_array(const struct bpf_map *map)
+{
+	return map->map_type == BPF_MAP_TYPE_INSN_ARRAY;
+}
+
+static inline bool valid_offsets(const struct bpf_insn_array *insn_array,
+				 const struct bpf_prog *prog)
+{
+	u32 off;
+	int i;
+
+	for (i = 0; i < insn_array->map.max_entries; i++) {
+		off = insn_array->ptrs[i].user_value.orig_off;
+
+		if (off >= prog->len)
+			return false;
+
+		if (off > 0) {
+			if (prog->insnsi[off-1].code == (BPF_LD | BPF_DW | BPF_IMM))
+				return false;
+		}
+	}
+
+	return true;
+}
+
+int bpf_insn_array_init(struct bpf_map *map, const struct bpf_prog *prog)
+{
+	struct bpf_insn_array *insn_array = cast_insn_array(map);
+	struct bpf_insn_array_value *val;
+	int i;
+
+	if (!is_frozen(map))
+		return -EINVAL;
+
+	if (!valid_offsets(insn_array, prog))
+		return -EINVAL;
+
+	/*
+	 * There can be only one program using the map
+	 */
+	if (atomic_xchg(&insn_array->used, 1))
+		return -EBUSY;
+
+	/*
+	 * Reset all the map indexes to the original values.  This is needed,
+	 * e.g., when a replay of verification with different log level should
+	 * be performed.
+	 */
+	for (i = 0; i < map->max_entries; i++) {
+		val = &insn_array->ptrs[i].user_value;
+		val->xlated_off = val->orig_off;
+	}
+
+	return 0;
+}
+
+int bpf_insn_array_ready(struct bpf_map *map)
+{
+	struct bpf_insn_array *insn_array = cast_insn_array(map);
+	int i;
+
+	for (i = 0; i < map->max_entries; i++) {
+		if (insn_array->ptrs[i].user_value.xlated_off == INSN_DELETED)
+			continue;
+		if (!insn_array->ips[i])
+			return -EFAULT;
+	}
+
+	return 0;
+}
+
+void bpf_insn_array_release(struct bpf_map *map)
+{
+	struct bpf_insn_array *insn_array = cast_insn_array(map);
+
+	atomic_set(&insn_array->used, 0);
+}
+
+void bpf_insn_array_adjust(struct bpf_map *map, u32 off, u32 len)
+{
+	struct bpf_insn_array *insn_array = cast_insn_array(map);
+	int i;
+
+	if (len <= 1)
+		return;
+
+	for (i = 0; i < map->max_entries; i++) {
+		if (insn_array->ptrs[i].user_value.xlated_off <= off)
+			continue;
+		if (insn_array->ptrs[i].user_value.xlated_off == INSN_DELETED)
+			continue;
+		insn_array->ptrs[i].user_value.xlated_off += len - 1;
+	}
+}
+
+void bpf_insn_array_adjust_after_remove(struct bpf_map *map, u32 off, u32 len)
+{
+	struct bpf_insn_array *insn_array = cast_insn_array(map);
+	int i;
+
+	for (i = 0; i < map->max_entries; i++) {
+		if (insn_array->ptrs[i].user_value.xlated_off < off)
+			continue;
+		if (insn_array->ptrs[i].user_value.xlated_off == INSN_DELETED)
+			continue;
+		if (insn_array->ptrs[i].user_value.xlated_off < off + len)
+			insn_array->ptrs[i].user_value.xlated_off = INSN_DELETED;
+		else
+			insn_array->ptrs[i].user_value.xlated_off -= len;
+	}
+}
+
+void bpf_prog_update_insn_ptr(struct bpf_prog *prog,
+			      u32 xlated_off,
+			      u32 jitted_off,
+			      void *jitted_ip)
+{
+	struct bpf_insn_array *insn_array;
+	struct bpf_map *map;
+	int i, j;
+
+	for (i = 0; i < prog->aux->used_map_cnt; i++) {
+		map = prog->aux->used_maps[i];
+		if (!is_insn_array(map))
+			continue;
+
+		insn_array = cast_insn_array(map);
+		for (j = 0; j < map->max_entries; j++) {
+			if (insn_array->ptrs[j].user_value.xlated_off == xlated_off) {
+				insn_array->ips[j] = (long)jitted_ip;
+				insn_array->ptrs[j].jitted_ip = jitted_ip;
+				insn_array->ptrs[j].user_value.jitted_off = jitted_off;
+			}
+		}
+	}
+}
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 8a129746bd6c..f62d61b6730a 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -1493,6 +1493,7 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr)
 	case BPF_MAP_TYPE_STRUCT_OPS:
 	case BPF_MAP_TYPE_CPUMAP:
 	case BPF_MAP_TYPE_ARENA:
+	case BPF_MAP_TYPE_INSN_ARRAY:
 		if (!bpf_token_capable(token, CAP_BPF))
 			goto put_token;
 		break;
@@ -2853,6 +2854,23 @@ static int bpf_prog_verify_signature(struct bpf_prog *prog, union bpf_attr *attr
 	return err;
 }
 
+static int bpf_prog_mark_insn_arrays_ready(struct bpf_prog *prog)
+{
+	int err;
+	int i;
+
+	for (i = 0; i < prog->aux->used_map_cnt; i++) {
+		if (prog->aux->used_maps[i]->map_type != BPF_MAP_TYPE_INSN_ARRAY)
+			continue;
+
+		err = bpf_insn_array_ready(prog->aux->used_maps[i]);
+		if (err)
+			return err;
+	}
+
+	return 0;
+}
+
 /* last field in 'union bpf_attr' used by this command */
 #define BPF_PROG_LOAD_LAST_FIELD keyring_id
 
@@ -3082,6 +3100,10 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size)
 	if (err < 0)
 		goto free_used_maps;
 
+	err = bpf_prog_mark_insn_arrays_ready(prog);
+	if (err < 0)
+		goto free_used_maps;
+
 	err = bpf_prog_alloc_id(prog);
 	if (err)
 		goto free_used_maps;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 4579082068ca..b4ad1f836c76 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -10083,6 +10083,8 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env,
 		    func_id != BPF_FUNC_map_push_elem)
 			goto error;
 		break;
+	case BPF_MAP_TYPE_INSN_ARRAY:
+		goto error;
 	default:
 		break;
 	}
@@ -20524,6 +20526,15 @@ static int __add_used_map(struct bpf_verifier_env *env, struct bpf_map *map)
 
 	env->used_maps[env->used_map_cnt++] = map;
 
+	if (map->map_type == BPF_MAP_TYPE_INSN_ARRAY) {
+		err = bpf_insn_array_init(map, env->prog);
+		if (err) {
+			verbose(env, "Failed to properly initialize insn array\n");
+			return err;
+		}
+		env->insn_array_maps[env->insn_array_map_cnt++] = map;
+	}
+
 	return env->used_map_cnt - 1;
 }
 
@@ -20770,6 +20781,33 @@ static void adjust_subprog_starts(struct bpf_verifier_env *env, u32 off, u32 len
 	}
 }
 
+static void release_insn_arrays(struct bpf_verifier_env *env)
+{
+	int i;
+
+	for (i = 0; i < env->insn_array_map_cnt; i++)
+		bpf_insn_array_release(env->insn_array_maps[i]);
+}
+
+static void adjust_insn_arrays(struct bpf_verifier_env *env, u32 off, u32 len)
+{
+	int i;
+
+	if (len == 1)
+		return;
+
+	for (i = 0; i < env->insn_array_map_cnt; i++)
+		bpf_insn_array_adjust(env->insn_array_maps[i], off, len);
+}
+
+static void adjust_insn_arrays_after_remove(struct bpf_verifier_env *env, u32 off, u32 len)
+{
+	int i;
+
+	for (i = 0; i < env->insn_array_map_cnt; i++)
+		bpf_insn_array_adjust_after_remove(env->insn_array_maps[i], off, len);
+}
+
 static void adjust_poke_descs(struct bpf_prog *prog, u32 off, u32 len)
 {
 	struct bpf_jit_poke_descriptor *tab = prog->aux->poke_tab;
@@ -20811,6 +20849,7 @@ static struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, u32 of
 	}
 	adjust_insn_aux_data(env, new_prog, off, len);
 	adjust_subprog_starts(env, off, len);
+	adjust_insn_arrays(env, off, len);
 	adjust_poke_descs(new_prog, off, len);
 	return new_prog;
 }
@@ -20994,6 +21033,8 @@ static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt)
 	if (err)
 		return err;
 
+	adjust_insn_arrays_after_remove(env, off, cnt);
+
 	memmove(aux_data + off,	aux_data + off + cnt,
 		sizeof(*aux_data) * (orig_prog_len - off - cnt));
 
@@ -24792,6 +24833,8 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr, __u3
 	adjust_btf_func(env);
 
 err_release_maps:
+	if (ret)
+		release_insn_arrays(env);
 	if (!env->prog->aux->used_maps)
 		/* if we didn't copy map pointers into bpf_prog_info, release
 		 * them now. Otherwise free_used_maps() will release them.
diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h
index 6829936d33f5..805d441363cd 100644
--- a/tools/include/uapi/linux/bpf.h
+++ b/tools/include/uapi/linux/bpf.h
@@ -1026,6 +1026,7 @@ enum bpf_map_type {
 	BPF_MAP_TYPE_USER_RINGBUF,
 	BPF_MAP_TYPE_CGRP_STORAGE,
 	BPF_MAP_TYPE_ARENA,
+	BPF_MAP_TYPE_INSN_ARRAY,
 	__MAX_BPF_MAP_TYPE
 };
 
@@ -7645,4 +7646,24 @@ enum bpf_kfunc_flags {
 	BPF_F_PAD_ZEROS = (1ULL << 0),
 };
 
+/*
+ * Values of a BPF_MAP_TYPE_INSN_ARRAY entry must be of this type.
+ *
+ * Before the map is used the orig_off field should point to an
+ * instruction inside the program being loaded. The other fields
+ * must be set to 0.
+ *
+ * After the program is loaded, the xlated_off will be adjusted
+ * by the verifier to point to the index of the original instruction
+ * in the xlated program. If the instruction is deleted, it will
+ * be set to (u32)-1. The jitted_off will be set to the corresponding
+ * offset in the jitted image of the program.
+ */
+struct bpf_insn_array_value {
+	__u32 orig_off;
+	__u32 xlated_off;
+	__u32 jitted_off;
+	__u32 :32;
+};
+
 #endif /* _UAPI__LINUX_BPF_H__ */
-- 
2.34.1

[PATCH v6 bpf-next 05/17] selftests/bpf: add selftests for new insn_array map

[Anton Protopopov]

Add the following selftests for new insn_array map:

  * Incorrect instruction indexes are rejected
  * Two programs can't use the same map
  * BPF progs can't operate the map
  * no changes to code => map is the same
  * expected changes when instructions are added
  * expected changes when instructions are deleted
  * expected changes when multiple functions are present

Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
---
 .../selftests/bpf/prog_tests/bpf_insn_array.c | 404 ++++++++++++++++++
 1 file changed, 404 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c

diff --git a/tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c b/tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c
new file mode 100644
index 000000000000..a4304ef5be13
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c
@@ -0,0 +1,404 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <bpf/bpf.h>
+#include <test_progs.h>
+
+#ifdef __x86_64__
+static int map_create(__u32 map_type, __u32 max_entries)
+{
+	const char *map_name = "insn_array";
+	__u32 key_size = 4;
+	__u32 value_size = sizeof(struct bpf_insn_array_value);
+
+	return bpf_map_create(map_type, map_name, key_size, value_size, max_entries, NULL);
+}
+
+static int prog_load(struct bpf_insn *insns, __u32 insn_cnt, int *fd_array, __u32 fd_array_cnt)
+{
+	LIBBPF_OPTS(bpf_prog_load_opts, opts);
+
+	opts.fd_array = fd_array;
+	opts.fd_array_cnt = fd_array_cnt;
+
+	return bpf_prog_load(BPF_PROG_TYPE_XDP, NULL, "GPL", insns, insn_cnt, &opts);
+}
+
+static void __check_success(struct bpf_insn *insns, __u32 insn_cnt, __u32 *map_in, __u32 *map_out)
+{
+	struct bpf_insn_array_value val = {};
+	int prog_fd = -1, map_fd, i;
+
+	map_fd = map_create(BPF_MAP_TYPE_INSN_ARRAY, insn_cnt);
+	if (!ASSERT_GE(map_fd, 0, "map_create"))
+		return;
+
+	for (i = 0; i < insn_cnt; i++) {
+		val.orig_off = map_in[i];
+		if (!ASSERT_EQ(bpf_map_update_elem(map_fd, &i, &val, 0), 0, "bpf_map_update_elem"))
+			goto cleanup;
+	}
+
+	if (!ASSERT_EQ(bpf_map_freeze(map_fd), 0, "bpf_map_freeze"))
+		goto cleanup;
+
+	prog_fd = prog_load(insns, insn_cnt, &map_fd, 1);
+	if (!ASSERT_GE(prog_fd, 0, "bpf(BPF_PROG_LOAD)"))
+		goto cleanup;
+
+	for (i = 0; i < insn_cnt; i++) {
+		char buf[64];
+
+		if (!ASSERT_EQ(bpf_map_lookup_elem(map_fd, &i, &val), 0, "bpf_map_lookup_elem"))
+			goto cleanup;
+
+		snprintf(buf, sizeof(buf), "val.xlated_off should be equal map_out[%d]", i);
+		ASSERT_EQ(val.xlated_off, map_out[i], buf);
+	}
+
+cleanup:
+	close(prog_fd);
+	close(map_fd);
+}
+
+/*
+ * Load a program, which will not be anyhow mangled by the verifier.  Add an
+ * insn_array map pointing to every instruction. Check that it hasn't changed
+ * after the program load.
+ */
+static void check_one_to_one_mapping(void)
+{
+	struct bpf_insn insns[] = {
+		BPF_MOV64_IMM(BPF_REG_0, 4),
+		BPF_MOV64_IMM(BPF_REG_0, 3),
+		BPF_MOV64_IMM(BPF_REG_0, 2),
+		BPF_MOV64_IMM(BPF_REG_0, 1),
+		BPF_MOV64_IMM(BPF_REG_0, 0),
+		BPF_EXIT_INSN(),
+	};
+	__u32 map_in[] = {0, 1, 2, 3, 4, 5};
+	__u32 map_out[] = {0, 1, 2, 3, 4, 5};
+
+	__check_success(insns, ARRAY_SIZE(insns), map_in, map_out);
+}
+
+/*
+ * Load a program with two patches (get jiffies, for simplicity). Add an
+ * insn_array map pointing to every instruction. Check how it was changed
+ * after the program load.
+ */
+static void check_simple(void)
+{
+	struct bpf_insn insns[] = {
+		BPF_MOV64_IMM(BPF_REG_0, 2),
+		BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_jiffies64),
+		BPF_MOV64_IMM(BPF_REG_0, 1),
+		BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_jiffies64),
+		BPF_MOV64_IMM(BPF_REG_0, 0),
+		BPF_EXIT_INSN(),
+	};
+	__u32 map_in[] = {0, 1, 2, 3, 4, 5};
+	__u32 map_out[] = {0, 1, 4, 5, 8, 9};
+
+	__check_success(insns, ARRAY_SIZE(insns), map_in, map_out);
+}
+
+/*
+ * Verifier can delete code in two cases: nops & dead code. From insn
+ * array's point of view, the two cases are the same, so test using
+ * the simplest method: by loading some nops
+ */
+static void check_deletions(void)
+{
+	struct bpf_insn insns[] = {
+		BPF_MOV64_IMM(BPF_REG_0, 2),
+		BPF_JMP_IMM(BPF_JA, 0, 0, 0), /* nop */
+		BPF_MOV64_IMM(BPF_REG_0, 1),
+		BPF_JMP_IMM(BPF_JA, 0, 0, 0), /* nop */
+		BPF_MOV64_IMM(BPF_REG_0, 0),
+		BPF_EXIT_INSN(),
+	};
+	__u32 map_in[] = {0, 1, 2, 3, 4, 5};
+	__u32 map_out[] = {0, -1, 1, -1, 2, 3};
+
+	__check_success(insns, ARRAY_SIZE(insns), map_in, map_out);
+}
+
+/*
+ * Same test as check_deletions, but also add code which adds instructions
+ */
+static void check_deletions_with_functions(void)
+{
+	struct bpf_insn insns[] = {
+		BPF_JMP_IMM(BPF_JA, 0, 0, 0), /* nop */
+		BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_jiffies64),
+		BPF_JMP_IMM(BPF_JA, 0, 0, 0), /* nop */
+		BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 1, 0, 2),
+		BPF_MOV64_IMM(BPF_REG_0, 1),
+		BPF_EXIT_INSN(),
+		BPF_JMP_IMM(BPF_JA, 0, 0, 0), /* nop */
+		BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_jiffies64),
+		BPF_JMP_IMM(BPF_JA, 0, 0, 0), /* nop */
+		BPF_MOV64_IMM(BPF_REG_0, 2),
+		BPF_EXIT_INSN(),
+	};
+	__u32 map_in[] =  { 0, 1,  2, 3, 4, 5, /* func */  6, 7,  8, 9, 10};
+	__u32 map_out[] = {-1, 0, -1, 3, 4, 5, /* func */ -1, 6, -1, 9, 10};
+
+	__check_success(insns, ARRAY_SIZE(insns), map_in, map_out);
+}
+
+/*
+ * Try to load a program with a map which points to outside of the program
+ */
+static void check_out_of_bounds_index(void)
+{
+	struct bpf_insn insns[] = {
+		BPF_MOV64_IMM(BPF_REG_0, 4),
+		BPF_MOV64_IMM(BPF_REG_0, 3),
+		BPF_MOV64_IMM(BPF_REG_0, 2),
+		BPF_MOV64_IMM(BPF_REG_0, 1),
+		BPF_MOV64_IMM(BPF_REG_0, 0),
+		BPF_EXIT_INSN(),
+	};
+	int prog_fd, map_fd;
+	struct bpf_insn_array_value val = {};
+	int key;
+
+	map_fd = map_create(BPF_MAP_TYPE_INSN_ARRAY, 1);
+	if (!ASSERT_GE(map_fd, 0, "map_create"))
+		return;
+
+	key = 0;
+	val.orig_off = ARRAY_SIZE(insns); /* too big */
+	if (!ASSERT_EQ(bpf_map_update_elem(map_fd, &key, &val, 0), 0, "bpf_map_update_elem"))
+		goto cleanup;
+
+	if (!ASSERT_EQ(bpf_map_freeze(map_fd), 0, "bpf_map_freeze"))
+		goto cleanup;
+
+	prog_fd = prog_load(insns, ARRAY_SIZE(insns), &map_fd, 1);
+	if (!ASSERT_EQ(prog_fd, -EINVAL, "program should have been rejected (prog_fd != -EINVAL)")) {
+		close(prog_fd);
+		goto cleanup;
+	}
+
+cleanup:
+	close(map_fd);
+}
+
+/*
+ * Try to load a program with a map which points to the middle of 16-bit insn
+ */
+static void check_mid_insn_index(void)
+{
+	struct bpf_insn insns[] = {
+		BPF_LD_IMM64(BPF_REG_0, 0), /* 2 x 8 */
+		BPF_EXIT_INSN(),
+	};
+	int prog_fd, map_fd;
+	struct bpf_insn_array_value val = {};
+	int key;
+
+	map_fd = map_create(BPF_MAP_TYPE_INSN_ARRAY, 1);
+	if (!ASSERT_GE(map_fd, 0, "map_create"))
+		return;
+
+	key = 0;
+	val.orig_off = 1; /* middle of 16-byte instruction */
+	if (!ASSERT_EQ(bpf_map_update_elem(map_fd, &key, &val, 0), 0, "bpf_map_update_elem"))
+		goto cleanup;
+
+	if (!ASSERT_EQ(bpf_map_freeze(map_fd), 0, "bpf_map_freeze"))
+		goto cleanup;
+
+	prog_fd = prog_load(insns, ARRAY_SIZE(insns), &map_fd, 1);
+	if (!ASSERT_EQ(prog_fd, -EINVAL, "program should have been rejected (prog_fd != -EINVAL)")) {
+		close(prog_fd);
+		goto cleanup;
+	}
+
+cleanup:
+	close(map_fd);
+}
+
+static void check_incorrect_index(void)
+{
+	check_out_of_bounds_index();
+	check_mid_insn_index();
+}
+
+/* Once map was initialized, it should be frozen */
+static void check_load_unfrozen_map(void)
+{
+	struct bpf_insn insns[] = {
+		BPF_MOV64_IMM(BPF_REG_0, 0),
+		BPF_EXIT_INSN(),
+	};
+	int prog_fd = -1, map_fd;
+	struct bpf_insn_array_value val = {};
+	int i;
+
+	map_fd = map_create(BPF_MAP_TYPE_INSN_ARRAY, ARRAY_SIZE(insns));
+	if (!ASSERT_GE(map_fd, 0, "map_create"))
+		return;
+
+	for (i = 0; i < ARRAY_SIZE(insns); i++) {
+		val.orig_off = i;
+		if (!ASSERT_EQ(bpf_map_update_elem(map_fd, &i, &val, 0), 0, "bpf_map_update_elem"))
+			goto cleanup;
+	}
+
+	prog_fd = prog_load(insns, ARRAY_SIZE(insns), &map_fd, 1);
+	if (!ASSERT_EQ(prog_fd, -EINVAL, "program should have been rejected (prog_fd != -EINVAL)"))
+		goto cleanup;
+
+	/* correctness: now freeze the map, the program should load fine */
+
+	if (!ASSERT_EQ(bpf_map_freeze(map_fd), 0, "bpf_map_freeze"))
+		goto cleanup;
+
+	prog_fd = prog_load(insns, ARRAY_SIZE(insns), &map_fd, 1);
+	if (!ASSERT_GE(prog_fd, 0, "bpf(BPF_PROG_LOAD)"))
+		goto cleanup;
+
+	for (i = 0; i < ARRAY_SIZE(insns); i++) {
+		if (!ASSERT_EQ(bpf_map_lookup_elem(map_fd, &i, &val), 0, "bpf_map_lookup_elem"))
+			goto cleanup;
+
+		ASSERT_EQ(val.xlated_off, i, "val should be equal i");
+	}
+
+cleanup:
+	close(prog_fd);
+	close(map_fd);
+}
+
+/* Map can be used only by one BPF program */
+static void check_no_map_reuse(void)
+{
+	struct bpf_insn insns[] = {
+		BPF_MOV64_IMM(BPF_REG_0, 0),
+		BPF_EXIT_INSN(),
+	};
+	int prog_fd = -1, map_fd, extra_fd = -1;
+	struct bpf_insn_array_value val = {};
+	int i;
+
+	map_fd = map_create(BPF_MAP_TYPE_INSN_ARRAY, ARRAY_SIZE(insns));
+	if (!ASSERT_GE(map_fd, 0, "map_create"))
+		return;
+
+	for (i = 0; i < ARRAY_SIZE(insns); i++) {
+		val.orig_off = i;
+		if (!ASSERT_EQ(bpf_map_update_elem(map_fd, &i, &val, 0), 0, "bpf_map_update_elem"))
+			goto cleanup;
+	}
+
+	if (!ASSERT_EQ(bpf_map_freeze(map_fd), 0, "bpf_map_freeze"))
+		goto cleanup;
+
+	prog_fd = prog_load(insns, ARRAY_SIZE(insns), &map_fd, 1);
+	if (!ASSERT_GE(prog_fd, 0, "bpf(BPF_PROG_LOAD)"))
+		goto cleanup;
+
+	for (i = 0; i < ARRAY_SIZE(insns); i++) {
+		if (!ASSERT_EQ(bpf_map_lookup_elem(map_fd, &i, &val), 0, "bpf_map_lookup_elem"))
+			goto cleanup;
+
+		ASSERT_EQ(val.xlated_off, i, "val should be equal i");
+	}
+
+	extra_fd = prog_load(insns, ARRAY_SIZE(insns), &map_fd, 1);
+	if (!ASSERT_EQ(extra_fd, -EBUSY, "program should have been rejected (extra_fd != -EBUSY)"))
+		goto cleanup;
+
+	/* correctness: check that prog is still loadable without fd_array */
+	extra_fd = prog_load(insns, ARRAY_SIZE(insns), NULL, 0);
+	if (!ASSERT_GE(prog_fd, 0, "bpf(BPF_PROG_LOAD): expected no error"))
+		goto cleanup;
+
+cleanup:
+	close(extra_fd);
+	close(prog_fd);
+	close(map_fd);
+}
+
+static void check_bpf_no_lookup(void)
+{
+	struct bpf_insn insns[] = {
+		BPF_LD_MAP_FD(BPF_REG_1, 0),
+		BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0),
+		BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+		BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8),
+		BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem),
+		BPF_EXIT_INSN(),
+	};
+	int prog_fd = -1, map_fd;
+
+	map_fd = map_create(BPF_MAP_TYPE_INSN_ARRAY, 1);
+	if (!ASSERT_GE(map_fd, 0, "map_create"))
+		return;
+
+	insns[0].imm = map_fd;
+
+	if (!ASSERT_EQ(bpf_map_freeze(map_fd), 0, "bpf_map_freeze"))
+		goto cleanup;
+
+	prog_fd = prog_load(insns, ARRAY_SIZE(insns), NULL, 0);
+	if (!ASSERT_EQ(prog_fd, -EINVAL, "program should have been rejected (prog_fd != -EINVAL)"))
+		goto cleanup;
+
+	/* correctness: check that prog is still loadable with normal map */
+	close(map_fd);
+	map_fd = map_create(BPF_MAP_TYPE_ARRAY, 1);
+	insns[0].imm = map_fd;
+	prog_fd = prog_load(insns, ARRAY_SIZE(insns), NULL, 0);
+	if (!ASSERT_GE(prog_fd, 0, "bpf(BPF_PROG_LOAD)"))
+		goto cleanup;
+
+cleanup:
+	close(prog_fd);
+	close(map_fd);
+}
+
+static void check_bpf_side(void)
+{
+	check_bpf_no_lookup();
+}
+
+void test_bpf_insn_array(void)
+{
+	/* Test if offsets are adjusted properly */
+
+	if (test__start_subtest("one2one"))
+		check_one_to_one_mapping();
+
+	if (test__start_subtest("simple"))
+		check_simple();
+
+	if (test__start_subtest("deletions"))
+		check_deletions();
+
+	if (test__start_subtest("deletions-with-functions"))
+		check_deletions_with_functions();
+
+	/* Check all kinds of operations and related restrictions */
+
+	if (test__start_subtest("incorrect-index"))
+		check_incorrect_index();
+
+	if (test__start_subtest("load-unfrozen-map"))
+		check_load_unfrozen_map();
+
+	if (test__start_subtest("no-map-reuse"))
+		check_no_map_reuse();
+
+	if (test__start_subtest("bpf-side-ops"))
+		check_bpf_side();
+}
+#else
+void test_bpf_insn_array(void)
+{
+
+}
+#endif
-- 
2.34.1

[PATCH v6 bpf-next 06/17] bpf: support instructions arrays with constants blinding

[Anton Protopopov]

When bpf_jit_harden is enabled, all constants in the BPF code are
blinded to prevent JIT spraying attacks. This happens during JIT
phase. Adjust all the related instruction arrays accordingly.

Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
Reviewed-by: Eduard Zingerman <eddyz87@gmail.com>
---
 kernel/bpf/core.c     | 20 ++++++++++++++++++++
 kernel/bpf/verifier.c | 11 ++++++++++-
 2 files changed, 30 insertions(+), 1 deletion(-)

diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index d595fe512498..4b62a03d6df5 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -1450,6 +1450,23 @@ void bpf_jit_prog_release_other(struct bpf_prog *fp, struct bpf_prog *fp_other)
 	bpf_prog_clone_free(fp_other);
 }
 
+static void adjust_insn_arrays(struct bpf_prog *prog, u32 off, u32 len)
+{
+#ifdef CONFIG_BPF_SYSCALL
+	struct bpf_map *map;
+	int i;
+
+	if (len <= 1)
+		return;
+
+	for (i = 0; i < prog->aux->used_map_cnt; i++) {
+		map = prog->aux->used_maps[i];
+		if (map->map_type == BPF_MAP_TYPE_INSN_ARRAY)
+			bpf_insn_array_adjust(map, off, len);
+	}
+#endif
+}
+
 struct bpf_prog *bpf_jit_blind_constants(struct bpf_prog *prog)
 {
 	struct bpf_insn insn_buff[16], aux[2];
@@ -1505,6 +1522,9 @@ struct bpf_prog *bpf_jit_blind_constants(struct bpf_prog *prog)
 		clone = tmp;
 		insn_delta = rewritten - 1;
 
+		/* Instructions arrays must be updated using absolute xlated offsets */
+		adjust_insn_arrays(clone, prog->aux->subprog_start + i, rewritten);
+
 		/* Walk new program and skip insns we just inserted. */
 		insn = clone->insnsi + i + insn_delta;
 		insn_cnt += insn_delta;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index b4ad1f836c76..4add3c778f02 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -21574,6 +21574,7 @@ static int jit_subprogs(struct bpf_verifier_env *env)
 	struct bpf_insn *insn;
 	void *old_bpf_func;
 	int err, num_exentries;
+	int old_len, subprog_start_adjustment = 0;
 
 	if (env->subprog_cnt <= 1)
 		return 0;
@@ -21648,7 +21649,7 @@ static int jit_subprogs(struct bpf_verifier_env *env)
 		func[i]->aux->func_idx = i;
 		/* Below members will be freed only at prog->aux */
 		func[i]->aux->btf = prog->aux->btf;
-		func[i]->aux->subprog_start = subprog_start;
+		func[i]->aux->subprog_start = subprog_start + subprog_start_adjustment;
 		func[i]->aux->func_info = prog->aux->func_info;
 		func[i]->aux->func_info_cnt = prog->aux->func_info_cnt;
 		func[i]->aux->poke_tab = prog->aux->poke_tab;
@@ -21702,7 +21703,15 @@ static int jit_subprogs(struct bpf_verifier_env *env)
 		func[i]->aux->might_sleep = env->subprog_info[i].might_sleep;
 		if (!i)
 			func[i]->aux->exception_boundary = env->seen_exception;
+
+		/*
+		 * To properly pass the absolute subprog start to jit
+		 * all instruction adjustments should be accumulated
+		 */
+		old_len = func[i]->len;
 		func[i] = bpf_int_jit_compile(func[i]);
+		subprog_start_adjustment += func[i]->len - old_len;
+
 		if (!func[i]->jited) {
 			err = -ENOTSUPP;
 			goto out_free;
-- 
2.34.1

[PATCH v6 bpf-next 07/17] selftests/bpf: test instructions arrays with blinding

[Anton Protopopov]

Add a specific test for instructions arrays with blinding enabled.

Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
---
 .../selftests/bpf/prog_tests/bpf_insn_array.c | 95 +++++++++++++++++++
 1 file changed, 95 insertions(+)

diff --git a/tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c b/tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c
index a4304ef5be13..bab55ae7687e 100644
--- a/tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c
+++ b/tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c
@@ -227,6 +227,98 @@ static void check_incorrect_index(void)
 	check_mid_insn_index();
 }
 
+static int set_bpf_jit_harden(char *level)
+{
+	char old_level;
+	int err = -1;
+	int fd = -1;
+
+	fd = open("/proc/sys/net/core/bpf_jit_harden", O_RDWR | O_NONBLOCK);
+	if (fd < 0) {
+		ASSERT_FAIL("open .../bpf_jit_harden returned %d (errno=%d)", fd, errno);
+		return -1;
+	}
+
+	err = read(fd, &old_level, 1);
+	if (err != 1) {
+		ASSERT_FAIL("read from .../bpf_jit_harden returned %d (errno=%d)", err, errno);
+		err = -1;
+		goto end;
+	}
+
+	lseek(fd, 0, SEEK_SET);
+
+	err = write(fd, level, 1);
+	if (err != 1) {
+		ASSERT_FAIL("write to .../bpf_jit_harden returned %d (errno=%d)", err, errno);
+		err = -1;
+		goto end;
+	}
+
+	err = 0;
+	*level = old_level;
+end:
+	if (fd >= 0)
+		close(fd);
+	return err;
+}
+
+static void check_blindness(void)
+{
+	struct bpf_insn insns[] = {
+		BPF_MOV64_IMM(BPF_REG_0, 4),
+		BPF_MOV64_IMM(BPF_REG_0, 3),
+		BPF_MOV64_IMM(BPF_REG_0, 2),
+		BPF_MOV64_IMM(BPF_REG_0, 1),
+		BPF_EXIT_INSN(),
+	};
+	int prog_fd = -1, map_fd;
+	struct bpf_insn_array_value val = {};
+	char bpf_jit_harden = '@'; /* non-exizsting value */
+	int i;
+
+	map_fd = map_create(BPF_MAP_TYPE_INSN_ARRAY, ARRAY_SIZE(insns));
+	if (!ASSERT_GE(map_fd, 0, "map_create"))
+		return;
+
+	for (i = 0; i < ARRAY_SIZE(insns); i++) {
+		val.orig_off = i;
+		if (!ASSERT_EQ(bpf_map_update_elem(map_fd, &i, &val, 0), 0, "bpf_map_update_elem"))
+			goto cleanup;
+	}
+
+	if (!ASSERT_EQ(bpf_map_freeze(map_fd), 0, "bpf_map_freeze"))
+		goto cleanup;
+
+	bpf_jit_harden = '2';
+	if (set_bpf_jit_harden(&bpf_jit_harden)) {
+		bpf_jit_harden = '@'; /* open, read or write failed => no write was done */
+		goto cleanup;
+	}
+
+	prog_fd = prog_load(insns, ARRAY_SIZE(insns), &map_fd, 1);
+	if (!ASSERT_GE(prog_fd, 0, "bpf(BPF_PROG_LOAD)"))
+		goto cleanup;
+
+	for (i = 0; i < ARRAY_SIZE(insns); i++) {
+		char fmt[32];
+
+		if (!ASSERT_EQ(bpf_map_lookup_elem(map_fd, &i, &val), 0, "bpf_map_lookup_elem"))
+			goto cleanup;
+
+		snprintf(fmt, sizeof(fmt), "val should be equal 3*%d", i);
+		ASSERT_EQ(val.xlated_off, i * 3, fmt);
+	}
+
+cleanup:
+	/* restore the old one */
+	if (bpf_jit_harden != '@')
+		set_bpf_jit_harden(&bpf_jit_harden);
+
+	close(prog_fd);
+	close(map_fd);
+}
+
 /* Once map was initialized, it should be frozen */
 static void check_load_unfrozen_map(void)
 {
@@ -382,6 +474,9 @@ void test_bpf_insn_array(void)
 	if (test__start_subtest("deletions-with-functions"))
 		check_deletions_with_functions();
 
+	if (test__start_subtest("blindness"))
+		check_blindness();
+
 	/* Check all kinds of operations and related restrictions */
 
 	if (test__start_subtest("incorrect-index"))
-- 
2.34.1

[PATCH v6 bpf-next 08/17] bpf, x86: allow indirect jumps to r8...r15

[Anton Protopopov]

Currently the emit_indirect_jump() function only accepts one of the
RAX, RCX, ..., RBP registers as the destination. Make it to accept
R8, R9, ..., R15 as well, and make callers to pass BPF registers, not
native registers. This is required to enable indirect jumps support
in eBPF.

Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
---
 arch/x86/net/bpf_jit_comp.c | 28 +++++++++++++++++++++-------
 1 file changed, 21 insertions(+), 7 deletions(-)

diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index c8e628410d2c..7443465ce9a4 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -660,24 +660,38 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type t,
 
 #define EMIT_LFENCE()	EMIT3(0x0F, 0xAE, 0xE8)
 
-static void emit_indirect_jump(u8 **pprog, int reg, u8 *ip)
+static void __emit_indirect_jump(u8 **pprog, int reg, bool ereg)
 {
 	u8 *prog = *pprog;
 
+	if (ereg)
+		EMIT1(0x41);
+
+	EMIT2(0xFF, 0xE0 + reg);
+
+	*pprog = prog;
+}
+
+static void emit_indirect_jump(u8 **pprog, int bpf_reg, u8 *ip)
+{
+	u8 *prog = *pprog;
+	int reg = reg2hex[bpf_reg];
+	bool ereg = is_ereg(bpf_reg);
+
 	if (cpu_feature_enabled(X86_FEATURE_INDIRECT_THUNK_ITS)) {
 		OPTIMIZER_HIDE_VAR(reg);
 		emit_jump(&prog, its_static_thunk(reg), ip);
 	} else if (cpu_feature_enabled(X86_FEATURE_RETPOLINE_LFENCE)) {
 		EMIT_LFENCE();
-		EMIT2(0xFF, 0xE0 + reg);
+		__emit_indirect_jump(&prog, reg, ereg);
 	} else if (cpu_feature_enabled(X86_FEATURE_RETPOLINE)) {
 		OPTIMIZER_HIDE_VAR(reg);
 		if (cpu_feature_enabled(X86_FEATURE_CALL_DEPTH))
-			emit_jump(&prog, &__x86_indirect_jump_thunk_array[reg], ip);
+			emit_jump(&prog, &__x86_indirect_jump_thunk_array[reg + 8*ereg], ip);
 		else
-			emit_jump(&prog, &__x86_indirect_thunk_array[reg], ip);
+			emit_jump(&prog, &__x86_indirect_thunk_array[reg + 8*ereg], ip);
 	} else {
-		EMIT2(0xFF, 0xE0 + reg);	/* jmp *%\reg */
+		__emit_indirect_jump(&prog, reg, ereg);
 		if (IS_ENABLED(CONFIG_MITIGATION_RETPOLINE) || IS_ENABLED(CONFIG_MITIGATION_SLS))
 			EMIT1(0xCC);		/* int3 */
 	}
@@ -797,7 +811,7 @@ static void emit_bpf_tail_call_indirect(struct bpf_prog *bpf_prog,
 	 * rdi == ctx (1st arg)
 	 * rcx == prog->bpf_func + X86_TAIL_CALL_OFFSET
 	 */
-	emit_indirect_jump(&prog, 1 /* rcx */, ip + (prog - start));
+	emit_indirect_jump(&prog, BPF_REG_4 /* R4 -> rcx */, ip + (prog - start));
 
 	/* out: */
 	ctx->tail_call_indirect_label = prog - start;
@@ -3551,7 +3565,7 @@ static int emit_bpf_dispatcher(u8 **pprog, int a, int b, s64 *progs, u8 *image,
 		if (err)
 			return err;
 
-		emit_indirect_jump(&prog, 2 /* rdx */, image + (prog - buf));
+		emit_indirect_jump(&prog, BPF_REG_3 /* R3 -> rdx */, image + (prog - buf));
 
 		*pprog = prog;
 		return 0;
-- 
2.34.1

[PATCH v6 bpf-next 09/17] bpf: make bpf_insn_successors to return a pointer

[Anton Protopopov]

The bpf_insn_successors() function is used to return successors
to a BPF instruction. So far, an instruction could have 0, 1 or 2
successors. Prepare the verifier code to introduction of instructions
with more than 2 successors (namely, indirect jumps).

To do this, introduce a new struct, struct bpf_iarray, containing
an array of bpf instruction indexes and make bpf_insn_successors
to return a pointer of that type. The storage for all instructions
is allocated in the env->succ, which holds an array of size 2,
to be used for all instructions.

Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
---
 include/linux/bpf_verifier.h | 12 +++++++-
 kernel/bpf/liveness.c        | 36 ++++++++++++++--------
 kernel/bpf/verifier.c        | 60 ++++++++++++++++++++++++------------
 3 files changed, 75 insertions(+), 33 deletions(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 142030e7e857..6b820d8d77af 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -509,6 +509,15 @@ struct bpf_map_ptr_state {
 #define BPF_ALU_SANITIZE		(BPF_ALU_SANITIZE_SRC | \
 					 BPF_ALU_SANITIZE_DST)
 
+/*
+ * An array of BPF instructions.
+ * Primary usage: return value of bpf_insn_successors.
+ */
+struct bpf_iarray {
+	int cnt;
+	u32 items[];
+};
+
 struct bpf_insn_aux_data {
 	union {
 		enum bpf_reg_type ptr_type;	/* pointer type for load/store insns */
@@ -830,6 +839,7 @@ struct bpf_verifier_env {
 	/* array of pointers to bpf_scc_info indexed by SCC id */
 	struct bpf_scc_info **scc_info;
 	u32 scc_cnt;
+	struct bpf_iarray *succ;
 };
 
 static inline struct bpf_func_info_aux *subprog_aux(struct bpf_verifier_env *env, int subprog)
@@ -1052,7 +1062,7 @@ void print_insn_state(struct bpf_verifier_env *env, const struct bpf_verifier_st
 
 struct bpf_subprog_info *bpf_find_containing_subprog(struct bpf_verifier_env *env, int off);
 int bpf_jmp_offset(struct bpf_insn *insn);
-int bpf_insn_successors(struct bpf_prog *prog, u32 idx, u32 succ[2]);
+struct bpf_iarray *bpf_insn_successors(struct bpf_verifier_env *env, u32 idx);
 void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask);
 bool bpf_calls_callback(struct bpf_verifier_env *env, int insn_idx);
 
diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index 1e6538f59a78..78fbde2d2b96 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -34,7 +34,7 @@
  *   - read and write marks propagation.
  * - The propagation phase is a textbook live variable data flow analysis:
  *
- *     state[cc, i].live_after = U [state[cc, s].live_before for s in insn_successors(i)]
+ *     state[cc, i].live_after = U [state[cc, s].live_before for s in bpf_insn_successors(i)]
  *     state[cc, i].live_before =
  *       (state[cc, i].live_after / state[cc, i].must_write) U state[i].may_read
  *
@@ -54,7 +54,7 @@
  *   The equation for "must_write_acc" propagation looks as follows:
  *
  *     state[cc, i].must_write_acc =
- *       ∩ [state[cc, s].must_write_acc for s in insn_successors(i)]
+ *       ∩ [state[cc, s].must_write_acc for s in bpf_insn_successors(i)]
  *       U state[cc, i].must_write
  *
  *   (An intersection of all "must_write_acc" for instruction successors
@@ -447,7 +447,12 @@ int bpf_jmp_offset(struct bpf_insn *insn)
 __diag_push();
 __diag_ignore_all("-Woverride-init", "Allow field initialization overrides for opcode_info_tbl");
 
-inline int bpf_insn_successors(struct bpf_prog *prog, u32 idx, u32 succ[2])
+/*
+ * Returns an array of instructions succ, with succ->items[0], ...,
+ * succ->items[n-1] with successor instructions, where n=succ->cnt
+ */
+inline struct bpf_iarray *
+bpf_insn_successors(struct bpf_verifier_env *env, u32 idx)
 {
 	static const struct opcode_info {
 		bool can_jump;
@@ -474,19 +479,25 @@ inline int bpf_insn_successors(struct bpf_prog *prog, u32 idx, u32 succ[2])
 		_J(BPF_JSET,  {.can_jump = true,  .can_fallthrough = true}),
 	#undef _J
 	};
+	struct bpf_prog *prog = env->prog;
 	struct bpf_insn *insn = &prog->insnsi[idx];
 	const struct opcode_info *opcode_info;
-	int i = 0, insn_sz;
+	struct bpf_iarray *succ;
+	int insn_sz;
+
+	/* pre-allocated array of size up to 2; reset cnt, as it may have been used already */
+	succ = env->succ;
+	succ->cnt = 0;
 
 	opcode_info = &opcode_info_tbl[BPF_CLASS(insn->code) | BPF_OP(insn->code)];
 	insn_sz = bpf_is_ldimm64(insn) ? 2 : 1;
 	if (opcode_info->can_fallthrough)
-		succ[i++] = idx + insn_sz;
+		succ->items[succ->cnt++] = idx + insn_sz;
 
 	if (opcode_info->can_jump)
-		succ[i++] = idx + bpf_jmp_offset(insn) + 1;
+		succ->items[succ->cnt++] = idx + bpf_jmp_offset(insn) + 1;
 
-	return i;
+	return succ;
 }
 
 __diag_pop();
@@ -546,11 +557,12 @@ static inline bool update_insn(struct bpf_verifier_env *env,
 	struct bpf_insn_aux_data *aux = env->insn_aux_data;
 	u64 new_before, new_after, must_write_acc;
 	struct per_frame_masks *insn, *succ_insn;
-	u32 succ_num, s, succ[2];
+	struct bpf_iarray *succ;
+	u32 s;
 	bool changed;
 
-	succ_num = bpf_insn_successors(env->prog, insn_idx, succ);
-	if (unlikely(succ_num == 0))
+	succ = bpf_insn_successors(env, insn_idx);
+	if (succ->cnt == 0)
 		return false;
 
 	changed = false;
@@ -562,8 +574,8 @@ static inline bool update_insn(struct bpf_verifier_env *env,
 	 * of successors plus all "must_write" slots of instruction itself.
 	 */
 	must_write_acc = U64_MAX;
-	for (s = 0; s < succ_num; ++s) {
-		succ_insn = get_frame_masks(instance, frame, succ[s]);
+	for (s = 0; s < succ->cnt; ++s) {
+		succ_insn = get_frame_masks(instance, frame, succ->items[s]);
 		new_after |= succ_insn->live_before;
 		must_write_acc &= succ_insn->must_write_acc;
 	}
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 4add3c778f02..ae017c032944 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -17807,6 +17807,22 @@ static int mark_fastcall_patterns(struct bpf_verifier_env *env)
 	return 0;
 }
 
+static struct bpf_iarray *iarray_realloc(struct bpf_iarray *old, size_t n_elem)
+{
+	size_t new_size = sizeof(struct bpf_iarray) + n_elem * sizeof(old->items[0]);
+	struct bpf_iarray *new;
+
+	new = kvrealloc(old, new_size, GFP_KERNEL_ACCOUNT);
+	if (!new) {
+		/* this is what callers always want, so simplify the call site */
+		kvfree(old);
+		return NULL;
+	}
+
+	new->cnt = n_elem;
+	return new;
+}
+
 /* Visits the instruction at index t and returns one of the following:
  *  < 0 - an error occurred
  *  DONE_EXPLORING - the instruction was fully explored
@@ -18027,8 +18043,9 @@ static int check_cfg(struct bpf_verifier_env *env)
  */
 static int compute_postorder(struct bpf_verifier_env *env)
 {
-	u32 cur_postorder, i, top, stack_sz, s, succ_cnt, succ[2];
+	u32 cur_postorder, i, top, stack_sz, s;
 	int *stack = NULL, *postorder = NULL, *state = NULL;
+	struct bpf_iarray *succ;
 
 	postorder = kvcalloc(env->prog->len, sizeof(int), GFP_KERNEL_ACCOUNT);
 	state = kvcalloc(env->prog->len, sizeof(int), GFP_KERNEL_ACCOUNT);
@@ -18052,11 +18069,11 @@ static int compute_postorder(struct bpf_verifier_env *env)
 				stack_sz--;
 				continue;
 			}
-			succ_cnt = bpf_insn_successors(env->prog, top, succ);
-			for (s = 0; s < succ_cnt; ++s) {
-				if (!state[succ[s]]) {
-					stack[stack_sz++] = succ[s];
-					state[succ[s]] |= DISCOVERED;
+			succ = bpf_insn_successors(env, top);
+			for (s = 0; s < succ->cnt; ++s) {
+				if (!state[succ->items[s]]) {
+					stack[stack_sz++] = succ->items[s];
+					state[succ->items[s]] |= DISCOVERED;
 				}
 			}
 			state[top] |= EXPLORED;
@@ -24363,14 +24380,13 @@ static int compute_live_registers(struct bpf_verifier_env *env)
 		for (i = 0; i < env->cfg.cur_postorder; ++i) {
 			int insn_idx = env->cfg.insn_postorder[i];
 			struct insn_live_regs *live = &state[insn_idx];
-			int succ_num;
-			u32 succ[2];
+			struct bpf_iarray *succ;
 			u16 new_out = 0;
 			u16 new_in = 0;
 
-			succ_num = bpf_insn_successors(env->prog, insn_idx, succ);
-			for (int s = 0; s < succ_num; ++s)
-				new_out |= state[succ[s]].in;
+			succ = bpf_insn_successors(env, insn_idx);
+			for (int s = 0; s < succ->cnt; ++s)
+				new_out |= state[succ->items[s]].in;
 			new_in = (new_out & ~live->def) | live->use;
 			if (new_out != live->out || new_in != live->in) {
 				live->in = new_in;
@@ -24423,11 +24439,11 @@ static int compute_scc(struct bpf_verifier_env *env)
 	const u32 insn_cnt = env->prog->len;
 	int stack_sz, dfs_sz, err = 0;
 	u32 *stack, *pre, *low, *dfs;
-	u32 succ_cnt, i, j, t, w;
+	u32 i, j, t, w;
 	u32 next_preorder_num;
 	u32 next_scc_id;
 	bool assign_scc;
-	u32 succ[2];
+	struct bpf_iarray *succ;
 
 	next_preorder_num = 1;
 	next_scc_id = 1;
@@ -24534,12 +24550,12 @@ static int compute_scc(struct bpf_verifier_env *env)
 				stack[stack_sz++] = w;
 			}
 			/* Visit 'w' successors */
-			succ_cnt = bpf_insn_successors(env->prog, w, succ);
-			for (j = 0; j < succ_cnt; ++j) {
-				if (pre[succ[j]]) {
-					low[w] = min(low[w], low[succ[j]]);
+			succ = bpf_insn_successors(env, w);
+			for (j = 0; j < succ->cnt; ++j) {
+				if (pre[succ->items[j]]) {
+					low[w] = min(low[w], low[succ->items[j]]);
 				} else {
-					dfs[dfs_sz++] = succ[j];
+					dfs[dfs_sz++] = succ->items[j];
 					goto dfs_continue;
 				}
 			}
@@ -24556,8 +24572,8 @@ static int compute_scc(struct bpf_verifier_env *env)
 			 * or if component has a self reference.
 			 */
 			assign_scc = stack[stack_sz - 1] != w;
-			for (j = 0; j < succ_cnt; ++j) {
-				if (succ[j] == w) {
+			for (j = 0; j < succ->cnt; ++j) {
+				if (succ->items[j] == w) {
 					assign_scc = true;
 					break;
 				}
@@ -24619,6 +24635,9 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr, __u3
 		goto err_free_env;
 	for (i = 0; i < len; i++)
 		env->insn_aux_data[i].orig_idx = i;
+	env->succ = iarray_realloc(NULL, 2);
+	if (!env->succ)
+		goto err_free_env;
 	env->prog = *prog;
 	env->ops = bpf_verifier_ops[env->prog->type];
 
@@ -24869,6 +24888,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr, __u3
 	bpf_stack_liveness_free(env);
 	kvfree(env->cfg.insn_postorder);
 	kvfree(env->scc_info);
+	kvfree(env->succ);
 	kvfree(env);
 	return ret;
 }
-- 
2.34.1

[PATCH v6 bpf-next 10/17] bpf, x86: add support for indirect jumps

[Anton Protopopov]

Add support for a new instruction

    BPF_JMP|BPF_X|BPF_JA, SRC=0, DST=Rx, off=0, imm=0

which does an indirect jump to a location stored in Rx.  The register
Rx should have type PTR_TO_INSN. This new type assures that the Rx
register contains a value (or a range of values) loaded from a
correct jump table – map of type instruction array.

For example, for a C switch LLVM will generate the following code:

    0:   r3 = r1                    # "switch (r3)"
    1:   if r3 > 0x13 goto +0x666   # check r3 boundaries
    2:   r3 <<= 0x3                 # adjust to an index in array of addresses
    3:   r1 = 0xbeef ll             # r1 is PTR_TO_MAP_VALUE, r1->map_ptr=M
    5:   r1 += r3                   # r1 inherits boundaries from r3
    6:   r1 = *(u64 *)(r1 + 0x0)    # r1 now has type INSN_TO_PTR
    7:   gotox r1                   # jit will generate proper code

Here the gotox instruction corresponds to one particular map. This is
possible however to have a gotox instruction which can be loaded from
different maps, e.g.

    0:   r1 &= 0x1
    1:   r2 <<= 0x3
    2:   r3 = 0x0 ll                # load from map M_1
    4:   r3 += r2
    5:   if r1 == 0x0 goto +0x4
    6:   r1 <<= 0x3
    7:   r3 = 0x0 ll                # load from map M_2
    9:   r3 += r1
    A:   r1 = *(u64 *)(r3 + 0x0)
    B:   gotox r1                   # jump to target loaded from M_1 or M_2

During check_cfg stage the verifier will collect all the maps which
point to inside the subprog being verified. When building the config,
the high 16 bytes of the insn_state are used, so this patch
(theoretically) supports jump tables of up to 2^16 slots.

During the later stage, in check_indirect_jump, it is checked that
the register Rx was loaded from a particular instruction array.

Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
---
 arch/x86/net/bpf_jit_comp.c  |   3 +
 include/linux/bpf.h          |   1 +
 include/linux/bpf_verifier.h |   9 +
 kernel/bpf/bpf_insn_array.c  |  15 ++
 kernel/bpf/core.c            |   1 +
 kernel/bpf/liveness.c        |   3 +
 kernel/bpf/log.c             |   1 +
 kernel/bpf/verifier.c        | 363 ++++++++++++++++++++++++++++++++++-
 8 files changed, 390 insertions(+), 6 deletions(-)

diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index 7443465ce9a4..b6e646039224 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -2629,6 +2629,9 @@ st:			if (is_imm8(insn->off))
 
 			break;
 
+		case BPF_JMP | BPF_JA | BPF_X:
+			emit_indirect_jump(&prog, insn->dst_reg, image + addrs[i - 1]);
+			break;
 		case BPF_JMP | BPF_JA:
 		case BPF_JMP32 | BPF_JA:
 			if (BPF_CLASS(insn->code) == BPF_JMP) {
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 363355628d2e..7c86043468e8 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -995,6 +995,7 @@ enum bpf_reg_type {
 	PTR_TO_ARENA,
 	PTR_TO_BUF,		 /* reg points to a read/write buffer */
 	PTR_TO_FUNC,		 /* reg points to a bpf program function */
+	PTR_TO_INSN,		 /* reg points to a bpf program instruction */
 	CONST_PTR_TO_DYNPTR,	 /* reg points to a const struct bpf_dynptr */
 	__BPF_REG_TYPE_MAX,
 
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 6b820d8d77af..5441341f1ab9 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -527,6 +527,7 @@ struct bpf_insn_aux_data {
 		struct {
 			u32 map_index;		/* index into used_maps[] */
 			u32 map_off;		/* offset from value base address */
+			struct bpf_iarray *jt;	/* jump table for gotox instruction */
 		};
 		struct {
 			enum bpf_reg_type reg_type;	/* type of pseudo_btf_id */
@@ -840,6 +841,7 @@ struct bpf_verifier_env {
 	struct bpf_scc_info **scc_info;
 	u32 scc_cnt;
 	struct bpf_iarray *succ;
+	struct bpf_iarray *gotox_tmp_buf;
 };
 
 static inline struct bpf_func_info_aux *subprog_aux(struct bpf_verifier_env *env, int subprog)
@@ -1050,6 +1052,13 @@ static inline bool bpf_stack_narrow_access_ok(int off, int fill_size, int spill_
 	return !(off % BPF_REG_SIZE);
 }
 
+static inline bool insn_is_gotox(struct bpf_insn *insn)
+{
+	return BPF_CLASS(insn->code) == BPF_JMP &&
+	       BPF_OP(insn->code) == BPF_JA &&
+	       BPF_SRC(insn->code) == BPF_X;
+}
+
 const char *reg_type_str(struct bpf_verifier_env *env, enum bpf_reg_type type);
 const char *dynptr_type_str(enum bpf_dynptr_type type);
 const char *iter_type_str(const struct btf *btf, u32 btf_id);
diff --git a/kernel/bpf/bpf_insn_array.c b/kernel/bpf/bpf_insn_array.c
index fa2c162a7264..71813ce734e9 100644
--- a/kernel/bpf/bpf_insn_array.c
+++ b/kernel/bpf/bpf_insn_array.c
@@ -127,6 +127,20 @@ static u64 insn_array_mem_usage(const struct bpf_map *map)
 	return insn_array_alloc_size(map->max_entries) + extra_size;
 }
 
+static int insn_array_map_direct_value_addr(const struct bpf_map *map, u64 *imm, u32 off)
+{
+	struct bpf_insn_array *insn_array = cast_insn_array(map);
+
+	if ((off % sizeof(long)) != 0 ||
+	    (off / sizeof(long)) >= map->max_entries)
+		return -EINVAL;
+
+	/* from BPF's point of view, this map is a jump table */
+	*imm = (unsigned long)insn_array->ips + off;
+
+	return 0;
+}
+
 BTF_ID_LIST_SINGLE(insn_array_btf_ids, struct, bpf_insn_array)
 
 const struct bpf_map_ops insn_array_map_ops = {
@@ -139,6 +153,7 @@ const struct bpf_map_ops insn_array_map_ops = {
 	.map_delete_elem = insn_array_delete_elem,
 	.map_check_btf = insn_array_check_btf,
 	.map_mem_usage = insn_array_mem_usage,
+	.map_direct_value_addr = insn_array_map_direct_value_addr,
 	.map_btf_id = &insn_array_btf_ids[0],
 };
 
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 4b62a03d6df5..ef4448f18aad 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -1708,6 +1708,7 @@ bool bpf_opcode_in_insntable(u8 code)
 		[BPF_LD | BPF_IND | BPF_B] = true,
 		[BPF_LD | BPF_IND | BPF_H] = true,
 		[BPF_LD | BPF_IND | BPF_W] = true,
+		[BPF_JMP | BPF_JA | BPF_X] = true,
 		[BPF_JMP | BPF_JCOND] = true,
 	};
 #undef BPF_INSN_3_TBL
diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index 78fbde2d2b96..b73e5c9ab61f 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -485,6 +485,9 @@ bpf_insn_successors(struct bpf_verifier_env *env, u32 idx)
 	struct bpf_iarray *succ;
 	int insn_sz;
 
+	if (unlikely(insn_is_gotox(insn)))
+		return env->insn_aux_data[idx].jt;
+
 	/* pre-allocated array of size up to 2; reset cnt, as it may have been used already */
 	succ = env->succ;
 	succ->cnt = 0;
diff --git a/kernel/bpf/log.c b/kernel/bpf/log.c
index f50533169cc3..3fb56617af36 100644
--- a/kernel/bpf/log.c
+++ b/kernel/bpf/log.c
@@ -461,6 +461,7 @@ const char *reg_type_str(struct bpf_verifier_env *env, enum bpf_reg_type type)
 		[PTR_TO_ARENA]		= "arena",
 		[PTR_TO_BUF]		= "buf",
 		[PTR_TO_FUNC]		= "func",
+		[PTR_TO_INSN]		= "insn",
 		[PTR_TO_MAP_KEY]	= "map_key",
 		[CONST_PTR_TO_DYNPTR]	= "dynptr_ptr",
 	};
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index ae017c032944..d2df21fde118 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -6003,6 +6003,18 @@ static int check_map_kptr_access(struct bpf_verifier_env *env, u32 regno,
 	return 0;
 }
 
+/*
+ * Return the size of the memory region accessible from a pointer to map value.
+ * For INSN_ARRAY maps whole bpf_insn_array->ips array is accessible.
+ */
+static u32 map_mem_size(const struct bpf_map *map)
+{
+	if (map->map_type == BPF_MAP_TYPE_INSN_ARRAY)
+		return map->max_entries * sizeof(long);
+
+	return map->value_size;
+}
+
 /* check read/write into a map element with possible variable offset */
 static int check_map_access(struct bpf_verifier_env *env, u32 regno,
 			    int off, int size, bool zero_size_allowed,
@@ -6012,11 +6024,11 @@ static int check_map_access(struct bpf_verifier_env *env, u32 regno,
 	struct bpf_func_state *state = vstate->frame[vstate->curframe];
 	struct bpf_reg_state *reg = &state->regs[regno];
 	struct bpf_map *map = reg->map_ptr;
+	u32 mem_size = map_mem_size(map);
 	struct btf_record *rec;
 	int err, i;
 
-	err = check_mem_region_access(env, regno, off, size, map->value_size,
-				      zero_size_allowed);
+	err = check_mem_region_access(env, regno, off, size, mem_size, zero_size_allowed);
 	if (err)
 		return err;
 
@@ -7478,6 +7490,8 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn
 {
 	struct bpf_reg_state *regs = cur_regs(env);
 	struct bpf_reg_state *reg = regs + regno;
+	bool insn_array = reg->type == PTR_TO_MAP_VALUE &&
+			  reg->map_ptr->map_type == BPF_MAP_TYPE_INSN_ARRAY;
 	int size, err = 0;
 
 	size = bpf_size_to_bytes(bpf_size);
@@ -7485,7 +7499,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn
 		return size;
 
 	/* alignment checks will add in reg->off themselves */
-	err = check_ptr_alignment(env, reg, off, size, strict_alignment_once);
+	err = check_ptr_alignment(env, reg, off, size, strict_alignment_once || insn_array);
 	if (err)
 		return err;
 
@@ -7512,6 +7526,11 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn
 			verbose(env, "R%d leaks addr into map\n", value_regno);
 			return -EACCES;
 		}
+		if (t == BPF_WRITE && insn_array) {
+			verbose(env, "writes into insn_array not allowed\n");
+			return -EACCES;
+		}
+
 		err = check_map_access_type(env, regno, off, size, t);
 		if (err)
 			return err;
@@ -7540,6 +7559,14 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn
 
 				regs[value_regno].type = SCALAR_VALUE;
 				__mark_reg_known(&regs[value_regno], val);
+			} else if (map->map_type == BPF_MAP_TYPE_INSN_ARRAY) {
+				if (bpf_size != BPF_DW) {
+					verbose(env, "Invalid read of %d bytes from insn_array\n",
+						     size);
+					return -EACCES;
+				}
+				copy_register_state(&regs[value_regno], reg);
+				regs[value_regno].type = PTR_TO_INSN;
 			} else {
 				mark_reg_unknown(env, regs, value_regno);
 			}
@@ -17055,7 +17082,8 @@ static int check_ld_imm(struct bpf_verifier_env *env, struct bpf_insn *insn)
 		}
 		dst_reg->type = PTR_TO_MAP_VALUE;
 		dst_reg->off = aux->map_off;
-		WARN_ON_ONCE(map->max_entries != 1);
+		WARN_ON_ONCE(map->map_type != BPF_MAP_TYPE_INSN_ARRAY &&
+			     map->max_entries != 1);
 		/* We want reg->id to be same (0) as map_value is not distinct */
 	} else if (insn->src_reg == BPF_PSEUDO_MAP_FD ||
 		   insn->src_reg == BPF_PSEUDO_MAP_IDX) {
@@ -17823,6 +17851,196 @@ static struct bpf_iarray *iarray_realloc(struct bpf_iarray *old, size_t n_elem)
 	return new;
 }
 
+static int copy_insn_array(struct bpf_map *map, u32 start, u32 end, u32 *items)
+{
+	struct bpf_insn_array_value *value;
+	u32 i;
+
+	for (i = start; i <= end; i++) {
+		value = map->ops->map_lookup_elem(map, &i);
+		if (!value)
+			return -EINVAL;
+		items[i - start] = value->xlated_off;
+	}
+	return 0;
+}
+
+static int cmp_ptr_to_u32(const void *a, const void *b)
+{
+	return *(u32 *)a - *(u32 *)b;
+}
+
+static int sort_insn_array_uniq(u32 *items, int cnt)
+{
+	int unique = 1;
+	int i;
+
+	sort(items, cnt, sizeof(items[0]), cmp_ptr_to_u32, NULL);
+
+	for (i = 1; i < cnt; i++)
+		if (items[i] != items[unique - 1])
+			items[unique++] = items[i];
+
+	return unique;
+}
+
+/*
+ * sort_unique({map[start], ..., map[end]}) into off
+ */
+static int copy_insn_array_uniq(struct bpf_map *map, u32 start, u32 end, u32 *off)
+{
+	u32 n = end - start + 1;
+	int err;
+
+	err = copy_insn_array(map, start, end, off);
+	if (err)
+		return err;
+
+	return sort_insn_array_uniq(off, n);
+}
+
+/*
+ * Copy all unique offsets from the map
+ */
+static struct bpf_iarray *jt_from_map(struct bpf_map *map)
+{
+	struct bpf_iarray *jt;
+	int n;
+
+	jt = iarray_realloc(NULL, map->max_entries);
+	if (!jt)
+		return ERR_PTR(-ENOMEM);
+
+	n = copy_insn_array_uniq(map, 0, map->max_entries - 1, jt->items);
+	if (n < 0) {
+		kvfree(jt);
+		return ERR_PTR(n);
+	}
+
+	return jt;
+}
+
+/*
+ * Find and collect all maps which fit in the subprog. Return the result as one
+ * combined jump table in jt->items (allocated with kvcalloc)
+ */
+static struct bpf_iarray *jt_from_subprog(struct bpf_verifier_env *env,
+					  int subprog_start, int subprog_end)
+{
+	struct bpf_iarray *jt = NULL;
+	struct bpf_map *map;
+	struct bpf_iarray *jt_cur;
+	int i;
+
+	for (i = 0; i < env->insn_array_map_cnt; i++) {
+		/*
+		 * TODO (when needed): collect only jump tables, not static keys
+		 * or maps for indirect calls
+		 */
+		map = env->insn_array_maps[i];
+
+		jt_cur = jt_from_map(map);
+		if (IS_ERR(jt_cur)) {
+			kvfree(jt);
+			return jt_cur;
+		}
+
+		/*
+		 * This is enough to check one element. The full table is
+		 * checked to fit inside the subprog later in create_jt()
+		 */
+		if (jt_cur->items[0] >= subprog_start && jt_cur->items[0] < subprog_end) {
+			u32 old_cnt = jt ? jt->cnt : 0;
+			jt = iarray_realloc(jt, old_cnt + jt_cur->cnt);
+			if (!jt) {
+				kvfree(jt_cur);
+				return ERR_PTR(-ENOMEM);
+			}
+			memcpy(jt->items + old_cnt, jt_cur->items, jt_cur->cnt << 2);
+		}
+
+		kvfree(jt_cur);
+	}
+
+	if (!jt) {
+		verbose(env, "no jump tables found for subprog starting at %u\n", subprog_start);
+		return ERR_PTR(-EINVAL);
+	}
+
+	jt->cnt = sort_insn_array_uniq(jt->items, jt->cnt);
+	return jt;
+}
+
+static struct bpf_iarray *
+create_jt(int t, struct bpf_verifier_env *env, int fd)
+{
+	static struct bpf_subprog_info *subprog;
+	int subprog_start, subprog_end;
+	struct bpf_iarray *jt;
+	int i;
+
+	subprog = bpf_find_containing_subprog(env, t);
+	subprog_start = subprog->start;
+	subprog_end = (subprog + 1)->start;
+	jt = jt_from_subprog(env, subprog_start, subprog_end);
+	if (IS_ERR(jt))
+		return jt;
+
+	/* Check that the every element of the jump table fits within the given subprogram */
+	for (i = 0; i < jt->cnt; i++) {
+		if (jt->items[i] < subprog_start || jt->items[i] >= subprog_end) {
+			verbose(env, "jump table for insn %d points outside of the subprog [%u,%u]",
+					t, subprog_start, subprog_end);
+			return ERR_PTR(-EINVAL);
+		}
+	}
+
+	return jt;
+}
+
+/* "conditional jump with N edges" */
+static int visit_gotox_insn(int t, struct bpf_verifier_env *env, int fd)
+{
+	int *insn_stack = env->cfg.insn_stack;
+	int *insn_state = env->cfg.insn_state;
+	bool keep_exploring = false;
+	struct bpf_iarray *jt;
+	int i, w;
+
+	jt = env->insn_aux_data[t].jt;
+	if (!jt) {
+		jt = create_jt(t, env, fd);
+		if (IS_ERR(jt))
+			return PTR_ERR(jt);
+
+		env->insn_aux_data[t].jt = jt;
+	}
+
+	mark_prune_point(env, t);
+	for (i = 0; i < jt->cnt; i++) {
+		w = jt->items[i];
+		if (w < 0 || w >= env->prog->len) {
+			verbose(env, "indirect jump out of range from insn %d to %d\n", t, w);
+			return -EINVAL;
+		}
+
+		mark_jmp_point(env, w);
+
+		/* EXPLORED || DISCOVERED */
+		if (insn_state[w])
+			continue;
+
+		if (env->cfg.cur_stack >= env->prog->len)
+			return -E2BIG;
+
+		insn_stack[env->cfg.cur_stack++] = w;
+		insn_state[w] |= DISCOVERED;
+		keep_exploring = true;
+	}
+
+	return keep_exploring ? KEEP_EXPLORING : DONE_EXPLORING;
+}
+
 /* Visits the instruction at index t and returns one of the following:
  *  < 0 - an error occurred
  *  DONE_EXPLORING - the instruction was fully explored
@@ -17915,8 +18133,8 @@ static int visit_insn(int t, struct bpf_verifier_env *env)
 		return visit_func_call_insn(t, insns, env, insn->src_reg == BPF_PSEUDO_CALL);
 
 	case BPF_JA:
-		if (BPF_SRC(insn->code) != BPF_K)
-			return -EINVAL;
+		if (BPF_SRC(insn->code) == BPF_X)
+			return visit_gotox_insn(t, env, insn->imm);
 
 		if (BPF_CLASS(insn->code) == BPF_JMP)
 			off = insn->off;
@@ -18845,6 +19063,9 @@ static bool regsafe(struct bpf_verifier_env *env, struct bpf_reg_state *rold,
 		return regs_exact(rold, rcur, idmap) && rold->frameno == rcur->frameno;
 	case PTR_TO_ARENA:
 		return true;
+	case PTR_TO_INSN:
+		return (rold->off == rcur->off && range_within(rold, rcur) &&
+			tnum_in(rold->var_off, rcur->var_off));
 	default:
 		return regs_exact(rold, rcur, idmap);
 	}
@@ -19854,6 +20075,99 @@ static int process_bpf_exit_full(struct bpf_verifier_env *env,
 	return PROCESS_BPF_EXIT;
 }
 
+static int indirect_jump_min_max_index(struct bpf_verifier_env *env,
+				       int regno,
+				       struct bpf_map *map,
+				       u32 *pmin_index, u32 *pmax_index)
+{
+	struct bpf_reg_state *reg = reg_state(env, regno);
+	u64 min_index, max_index;
+	const u32 size = 8;
+
+	if (check_add_overflow(reg->umin_value, reg->off, &min_index) ||
+		(min_index > (u64) U32_MAX * size)) {
+		verbose(env, "the sum of R%u umin_value %llu and off %u is too big\n",
+			     regno, reg->umin_value, reg->off);
+		return -ERANGE;
+	}
+	if (check_add_overflow(reg->umax_value, reg->off, &max_index) ||
+		(max_index > (u64) U32_MAX * size)) {
+		verbose(env, "the sum of R%u umax_value %llu and off %u is too big\n",
+			     regno, reg->umax_value, reg->off);
+		return -ERANGE;
+	}
+
+	min_index /= size;
+	max_index /= size;
+
+	if (max_index >= map->max_entries) {
+		verbose(env, "R%u points to outside of jump table: [%llu,%llu] max_entries %u\n",
+			     regno, min_index, max_index, map->max_entries);
+		return -EINVAL;
+	}
+
+	*pmin_index = min_index;
+	*pmax_index = max_index;
+	return 0;
+}
+
+/* gotox *dst_reg */
+static int check_indirect_jump(struct bpf_verifier_env *env, struct bpf_insn *insn)
+{
+	struct bpf_verifier_state *other_branch;
+	struct bpf_reg_state *dst_reg;
+	struct bpf_map *map;
+	u32 min_index, max_index;
+	int err = 0;
+	int n;
+	int i;
+
+	dst_reg = reg_state(env, insn->dst_reg);
+	if (dst_reg->type != PTR_TO_INSN) {
+		verbose(env, "R%d has type %d, expected PTR_TO_INSN\n",
+			     insn->dst_reg, dst_reg->type);
+		return -EINVAL;
+	}
+
+	map = dst_reg->map_ptr;
+	if (verifier_bug_if(!map, env, "R%d has an empty map pointer", insn->dst_reg))
+		return -EFAULT;
+
+	if (verifier_bug_if(map->map_type != BPF_MAP_TYPE_INSN_ARRAY, env,
+			    "R%d has incorrect map type %d", insn->dst_reg, map->map_type))
+		return -EFAULT;
+
+	err = indirect_jump_min_max_index(env, insn->dst_reg, map, &min_index, &max_index);
+	if (err)
+		return err;
+
+	/* Ensure that the buffer is large enough */
+	if (!env->gotox_tmp_buf || env->gotox_tmp_buf->cnt < max_index - min_index + 1) {
+		env->gotox_tmp_buf = iarray_realloc(env->gotox_tmp_buf,
+						    max_index - min_index + 1);
+		if (!env->gotox_tmp_buf)
+			return -ENOMEM;
+	}
+
+	n = copy_insn_array_uniq(map, min_index, max_index, env->gotox_tmp_buf->items);
+	if (n < 0)
+		return n;
+	if (n == 0) {
+		verbose(env, "register R%d doesn't point to any offset in map id=%d\n",
+			     insn->dst_reg, map->id);
+		return -EINVAL;
+	}
+
+	for (i = 0; i < n - 1; i++) {
+		other_branch = push_stack(env, env->gotox_tmp_buf->items[i],
+					  env->insn_idx, env->cur_state->speculative);
+		if (IS_ERR(other_branch))
+			return PTR_ERR(other_branch);
+	}
+	env->insn_idx = env->gotox_tmp_buf->items[n-1];
+	return 0;
+}
+
 static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state)
 {
 	int err;
@@ -19956,6 +20270,15 @@ static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state)
 
 			mark_reg_scratched(env, BPF_REG_0);
 		} else if (opcode == BPF_JA) {
+			if (BPF_SRC(insn->code) == BPF_X) {
+				if (insn->src_reg != BPF_REG_0 ||
+				    insn->imm != 0 || insn->off != 0) {
+					verbose(env, "BPF_JA|BPF_X uses reserved fields\n");
+					return -EINVAL;
+				}
+				return check_indirect_jump(env, insn);
+			}
+
 			if (BPF_SRC(insn->code) != BPF_K ||
 			    insn->src_reg != BPF_REG_0 ||
 			    insn->dst_reg != BPF_REG_0 ||
@@ -20472,6 +20795,7 @@ static int check_map_prog_compatibility(struct bpf_verifier_env *env,
 		case BPF_MAP_TYPE_QUEUE:
 		case BPF_MAP_TYPE_STACK:
 		case BPF_MAP_TYPE_ARENA:
+		case BPF_MAP_TYPE_INSN_ARRAY:
 			break;
 		default:
 			verbose(env,
@@ -21029,6 +21353,27 @@ static int bpf_adj_linfo_after_remove(struct bpf_verifier_env *env, u32 off,
 	return 0;
 }
 
+/*
+ * Clean up dynamically allocated fields of aux data for instructions [start, ...]
+ */
+static void clear_insn_aux_data(struct bpf_verifier_env *env, int start, int len)
+{
+	struct bpf_insn_aux_data *aux_data = env->insn_aux_data;
+	struct bpf_insn *insns = env->prog->insnsi;
+	int end = start + len;
+	int i;
+
+	for (i = start; i < end; i++) {
+		if (insn_is_gotox(&insns[i])) {
+			kvfree(aux_data[i].jt);
+			aux_data[i].jt = NULL;
+		}
+
+		if (bpf_is_ldimm64(&insns[i]))
+			i++;
+	}
+}
+
 static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt)
 {
 	struct bpf_insn_aux_data *aux_data = env->insn_aux_data;
@@ -21052,6 +21397,8 @@ static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt)
 
 	adjust_insn_arrays_after_remove(env, off, cnt);
 
+	clear_insn_aux_data(env, off, cnt);
+
 	memmove(aux_data + off,	aux_data + off + cnt,
 		sizeof(*aux_data) * (orig_prog_len - off - cnt));
 
@@ -21696,6 +22043,8 @@ static int jit_subprogs(struct bpf_verifier_env *env)
 		func[i]->aux->jited_linfo = prog->aux->jited_linfo;
 		func[i]->aux->linfo_idx = env->subprog_info[i].linfo_idx;
 		func[i]->aux->arena = prog->aux->arena;
+		func[i]->aux->used_maps = env->used_maps;
+		func[i]->aux->used_map_cnt = env->used_map_cnt;
 		num_exentries = 0;
 		insn = func[i]->insnsi;
 		for (j = 0; j < func[i]->len; j++, insn++) {
@@ -24883,12 +25232,14 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr, __u3
 err_unlock:
 	if (!is_priv)
 		mutex_unlock(&bpf_verifier_lock);
+	clear_insn_aux_data(env, 0, env->prog->len);
 	vfree(env->insn_aux_data);
 err_free_env:
 	bpf_stack_liveness_free(env);
 	kvfree(env->cfg.insn_postorder);
 	kvfree(env->scc_info);
 	kvfree(env->succ);
+	kvfree(env->gotox_tmp_buf);
 	kvfree(env);
 	return ret;
 }
-- 
2.34.1

[PATCH v6 bpf-next 11/17] bpf: disasm: add support for BPF_JMP|BPF_JA|BPF_X

[Anton Protopopov]

Add support for indirect jump instruction.

Example output from bpftool:

   0: (79) r3 = *(u64 *)(r1 +0)
   1: (25) if r3 > 0x4 goto pc+666
   2: (67) r3 <<= 3
   3: (18) r1 = 0xffffbeefspameggs
   5: (0f) r1 += r3
   6: (79) r1 = *(u64 *)(r1 +0)
   7: (0d) gotox r1

Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
---
 kernel/bpf/disasm.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/kernel/bpf/disasm.c b/kernel/bpf/disasm.c
index 20883c6b1546..f8a3c7eb451e 100644
--- a/kernel/bpf/disasm.c
+++ b/kernel/bpf/disasm.c
@@ -358,6 +358,9 @@ void print_bpf_insn(const struct bpf_insn_cbs *cbs,
 		} else if (insn->code == (BPF_JMP | BPF_JA)) {
 			verbose(cbs->private_data, "(%02x) goto pc%+d\n",
 				insn->code, insn->off);
+		} else if (insn->code == (BPF_JMP | BPF_JA | BPF_X)) {
+			verbose(cbs->private_data, "(%02x) gotox r%d\n",
+				insn->code, insn->dst_reg);
 		} else if (insn->code == (BPF_JMP | BPF_JCOND) &&
 			   insn->src_reg == BPF_MAY_GOTO) {
 			verbose(cbs->private_data, "(%02x) may_goto pc%+d\n",
-- 
2.34.1

[PATCH v6 bpf-next 12/17] bpf, docs: do not state that indirect jumps are not supported

[Anton Protopopov]

The linux-notes.rst states that indirect jump instruction "is not
currently supported by the verifier". Remove this part as outdated.

Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
---
 Documentation/bpf/linux-notes.rst | 8 --------
 1 file changed, 8 deletions(-)

diff --git a/Documentation/bpf/linux-notes.rst b/Documentation/bpf/linux-notes.rst
index 00d2693de025..64ac146a926f 100644
--- a/Documentation/bpf/linux-notes.rst
+++ b/Documentation/bpf/linux-notes.rst
@@ -12,14 +12,6 @@ Byte swap instructions
 
 ``BPF_FROM_LE`` and ``BPF_FROM_BE`` exist as aliases for ``BPF_TO_LE`` and ``BPF_TO_BE`` respectively.
 
-Jump instructions
-=================
-
-``BPF_CALL | BPF_X | BPF_JMP`` (0x8d), where the helper function
-integer would be read from a specified register, is not currently supported
-by the verifier.  Any programs with this instruction will fail to load
-until such support is added.
-
 Maps
 ====
 
-- 
2.34.1

[PATCH v6 bpf-next 13/17] libbpf: fix formatting of bpf_object__append_subprog_code

[Anton Protopopov]

The commit 6c918709bd30 ("libbpf: Refactor bpf_object__reloc_code")
added the bpf_object__append_subprog_code() with incorrect indentations.
Use tabs instead. (This also makes a consequent commit better readable.)

Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
Acked-by: Andrii Nakryiko <andrii@kernel.org>
---
 tools/lib/bpf/libbpf.c | 52 +++++++++++++++++++++---------------------
 1 file changed, 26 insertions(+), 26 deletions(-)

diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
index dd3b2f57082d..b90574f39d1c 100644
--- a/tools/lib/bpf/libbpf.c
+++ b/tools/lib/bpf/libbpf.c
@@ -6436,32 +6436,32 @@ static int
 bpf_object__append_subprog_code(struct bpf_object *obj, struct bpf_program *main_prog,
 				struct bpf_program *subprog)
 {
-       struct bpf_insn *insns;
-       size_t new_cnt;
-       int err;
-
-       subprog->sub_insn_off = main_prog->insns_cnt;
-
-       new_cnt = main_prog->insns_cnt + subprog->insns_cnt;
-       insns = libbpf_reallocarray(main_prog->insns, new_cnt, sizeof(*insns));
-       if (!insns) {
-               pr_warn("prog '%s': failed to realloc prog code\n", main_prog->name);
-               return -ENOMEM;
-       }
-       main_prog->insns = insns;
-       main_prog->insns_cnt = new_cnt;
-
-       memcpy(main_prog->insns + subprog->sub_insn_off, subprog->insns,
-              subprog->insns_cnt * sizeof(*insns));
-
-       pr_debug("prog '%s': added %zu insns from sub-prog '%s'\n",
-                main_prog->name, subprog->insns_cnt, subprog->name);
-
-       /* The subprog insns are now appended. Append its relos too. */
-       err = append_subprog_relos(main_prog, subprog);
-       if (err)
-               return err;
-       return 0;
+	struct bpf_insn *insns;
+	size_t new_cnt;
+	int err;
+
+	subprog->sub_insn_off = main_prog->insns_cnt;
+
+	new_cnt = main_prog->insns_cnt + subprog->insns_cnt;
+	insns = libbpf_reallocarray(main_prog->insns, new_cnt, sizeof(*insns));
+	if (!insns) {
+		pr_warn("prog '%s': failed to realloc prog code\n", main_prog->name);
+		return -ENOMEM;
+	}
+	main_prog->insns = insns;
+	main_prog->insns_cnt = new_cnt;
+
+	memcpy(main_prog->insns + subprog->sub_insn_off, subprog->insns,
+	       subprog->insns_cnt * sizeof(*insns));
+
+	pr_debug("prog '%s': added %zu insns from sub-prog '%s'\n",
+		 main_prog->name, subprog->insns_cnt, subprog->name);
+
+	/* The subprog insns are now appended. Append its relos too. */
+	err = append_subprog_relos(main_prog, subprog);
+	if (err)
+		return err;
+	return 0;
 }
 
 static int
-- 
2.34.1

[PATCH v6 bpf-next 14/17] libbpf: support llvm-generated indirect jumps

[Anton Protopopov]

For v4 instruction set LLVM is allowed to generate indirect jumps for
switch statements and for 'goto *rX' assembly. Every such a jump will
be accompanied by necessary metadata, e.g. (`llvm-objdump -Sr ...`):

       0:       r2 = 0x0 ll
                0000000000000030:  R_BPF_64_64  BPF.JT.0.0

Here BPF.JT.1.0 is a symbol residing in the .jumptables section:

    Symbol table:
       4: 0000000000000000   240 OBJECT  GLOBAL DEFAULT     4 BPF.JT.0.0

The -bpf-min-jump-table-entries llvm option may be used to control the
minimal size of a switch which will be converted to an indirect jumps.

Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
---
 tools/lib/bpf/libbpf.c        | 240 +++++++++++++++++++++++++++++++++-
 tools/lib/bpf/libbpf_probes.c |   4 +
 tools/lib/bpf/linker.c        |  10 +-
 3 files changed, 251 insertions(+), 3 deletions(-)

diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
index b90574f39d1c..ee44bc49a3ba 100644
--- a/tools/lib/bpf/libbpf.c
+++ b/tools/lib/bpf/libbpf.c
@@ -190,6 +190,7 @@ static const char * const map_type_name[] = {
 	[BPF_MAP_TYPE_USER_RINGBUF]             = "user_ringbuf",
 	[BPF_MAP_TYPE_CGRP_STORAGE]		= "cgrp_storage",
 	[BPF_MAP_TYPE_ARENA]			= "arena",
+	[BPF_MAP_TYPE_INSN_ARRAY]		= "insn_array",
 };
 
 static const char * const prog_type_name[] = {
@@ -369,6 +370,7 @@ enum reloc_type {
 	RELO_EXTERN_CALL,
 	RELO_SUBPROG_ADDR,
 	RELO_CORE,
+	RELO_INSN_ARRAY,
 };
 
 struct reloc_desc {
@@ -379,7 +381,16 @@ struct reloc_desc {
 		struct {
 			int map_idx;
 			int sym_off;
-			int ext_idx;
+			/*
+			 * The following two fields can be unionized, as the
+			 * ext_idx field is used for extern symbols, and the
+			 * sym_size is used for jump tables, which are never
+			 * extern
+			 */
+			union {
+				int ext_idx;
+				int sym_size;
+			};
 		};
 	};
 };
@@ -421,6 +432,11 @@ struct bpf_sec_def {
 	libbpf_prog_attach_fn_t prog_attach_fn;
 };
 
+struct bpf_light_subprog {
+	__u32 sec_insn_off;
+	__u32 sub_insn_off;
+};
+
 /*
  * bpf_prog should be a better name but it has been used in
  * linux/filter.h.
@@ -494,6 +510,9 @@ struct bpf_program {
 	__u32 line_info_cnt;
 	__u32 prog_flags;
 	__u8  hash[SHA256_DIGEST_LENGTH];
+
+	struct bpf_light_subprog *subprogs;
+	__u32 subprog_cnt;
 };
 
 struct bpf_struct_ops {
@@ -523,6 +542,7 @@ struct bpf_struct_ops {
 #define STRUCT_OPS_SEC ".struct_ops"
 #define STRUCT_OPS_LINK_SEC ".struct_ops.link"
 #define ARENA_SEC ".addr_space.1"
+#define JUMPTABLES_SEC ".jumptables"
 
 enum libbpf_map_type {
 	LIBBPF_MAP_UNSPEC,
@@ -667,6 +687,7 @@ struct elf_state {
 	int symbols_shndx;
 	bool has_st_ops;
 	int arena_data_shndx;
+	int jumptables_data_shndx;
 };
 
 struct usdt_manager;
@@ -738,6 +759,16 @@ struct bpf_object {
 	void *arena_data;
 	size_t arena_data_sz;
 
+	void *jumptables_data;
+	size_t jumptables_data_sz;
+
+	struct {
+		struct bpf_program *prog;
+		int sym_off;
+		int fd;
+	} *jumptable_maps;
+	size_t jumptable_map_cnt;
+
 	struct kern_feature_cache *feat_cache;
 	char *token_path;
 	int token_fd;
@@ -764,6 +795,7 @@ void bpf_program__unload(struct bpf_program *prog)
 
 	zfree(&prog->func_info);
 	zfree(&prog->line_info);
+	zfree(&prog->subprogs);
 }
 
 static void bpf_program__exit(struct bpf_program *prog)
@@ -3942,6 +3974,13 @@ static int bpf_object__elf_collect(struct bpf_object *obj)
 			} else if (strcmp(name, ARENA_SEC) == 0) {
 				obj->efile.arena_data = data;
 				obj->efile.arena_data_shndx = idx;
+			} else if (strcmp(name, JUMPTABLES_SEC) == 0) {
+				obj->jumptables_data = malloc(data->d_size);
+				if (!obj->jumptables_data)
+					return -ENOMEM;
+				memcpy(obj->jumptables_data, data->d_buf, data->d_size);
+				obj->jumptables_data_sz = data->d_size;
+				obj->efile.jumptables_data_shndx = idx;
 			} else {
 				pr_info("elf: skipping unrecognized data section(%d) %s\n",
 					idx, name);
@@ -4634,6 +4673,16 @@ static int bpf_program__record_reloc(struct bpf_program *prog,
 		return 0;
 	}
 
+	/* jump table data relocation */
+	if (shdr_idx == obj->efile.jumptables_data_shndx) {
+		reloc_desc->type = RELO_INSN_ARRAY;
+		reloc_desc->insn_idx = insn_idx;
+		reloc_desc->map_idx = -1;
+		reloc_desc->sym_off = sym->st_value;
+		reloc_desc->sym_size = sym->st_size;
+		return 0;
+	}
+
 	/* generic map reference relocation */
 	if (type == LIBBPF_MAP_UNSPEC) {
 		if (!bpf_object__shndx_is_maps(obj, shdr_idx)) {
@@ -6144,6 +6193,144 @@ static void poison_kfunc_call(struct bpf_program *prog, int relo_idx,
 	insn->imm = POISON_CALL_KFUNC_BASE + ext_idx;
 }
 
+static int find_jt_map(struct bpf_object *obj, struct bpf_program *prog, int sym_off)
+{
+	size_t i;
+
+	for (i = 0; i < obj->jumptable_map_cnt; i++) {
+		/*
+		 * This might happen that same offset is used for two different
+		 * programs (as jump tables can be the same). However, for
+		 * different programs different maps should be created.
+		 */
+		if (obj->jumptable_maps[i].sym_off == sym_off &&
+		    obj->jumptable_maps[i].prog == prog)
+			return obj->jumptable_maps[i].fd;
+	}
+
+	return -ENOENT;
+}
+
+static int add_jt_map(struct bpf_object *obj, struct bpf_program *prog, int sym_off, int map_fd)
+{
+	size_t new_cnt = obj->jumptable_map_cnt + 1;
+	size_t size = sizeof(obj->jumptable_maps[0]);
+	void *tmp;
+
+	tmp = libbpf_reallocarray(obj->jumptable_maps, new_cnt, size);
+	if (!tmp)
+		return -ENOMEM;
+
+	obj->jumptable_maps = tmp;
+	obj->jumptable_maps[new_cnt - 1].prog = prog;
+	obj->jumptable_maps[new_cnt - 1].sym_off = sym_off;
+	obj->jumptable_maps[new_cnt - 1].fd = map_fd;
+	obj->jumptable_map_cnt = new_cnt;
+
+	return 0;
+}
+
+static int create_jt_map(struct bpf_object *obj, struct bpf_program *prog,
+			 int sym_off, int jt_size, int adjust_off)
+{
+	const __u32 jt_entry_size = 8;
+	const __u32 max_entries = jt_size / jt_entry_size;
+	const __u32 value_size = sizeof(struct bpf_insn_array_value);
+	struct bpf_insn_array_value val = {};
+	int map_fd, err;
+	__u64 insn_off;
+	__u64 *jt;
+	__u32 i;
+
+	map_fd = find_jt_map(obj, prog, sym_off);
+	if (map_fd >= 0)
+		return map_fd;
+
+	if (sym_off % jt_entry_size) {
+		pr_warn("jumptable start %d should be multiple of %u\n",
+			sym_off, jt_entry_size);
+		return -EINVAL;
+	}
+
+	if (jt_size % jt_entry_size) {
+		pr_warn("jumptable size %d should be multiple of %u\n",
+			jt_size, jt_entry_size);
+		return -EINVAL;
+	}
+
+	map_fd = bpf_map_create(BPF_MAP_TYPE_INSN_ARRAY, ".jumptables",
+				4, value_size, max_entries, NULL);
+	if (map_fd < 0)
+		return map_fd;
+
+	if (!obj->jumptables_data) {
+		pr_warn("map '.jumptables': ELF file is missing jump table data\n");
+		err = -EINVAL;
+		goto err_close;
+	}
+	if (sym_off + jt_size > obj->jumptables_data_sz) {
+		pr_warn("jumptables_data size is %zd, trying to access %d\n",
+			obj->jumptables_data_sz, sym_off + jt_size);
+		err = -EINVAL;
+		goto err_close;
+	}
+
+	jt = (__u64 *)(obj->jumptables_data + sym_off);
+	for (i = 0; i < max_entries; i++) {
+		/*
+		 * LLVM-generated jump tables contain u64 records, however
+		 * should contain values that fit in u32.
+		 * The adjust_off provided by the caller adjusts the offset to
+		 * be relative to the beginning of the main function
+		 */
+		insn_off = jt[i]/sizeof(struct bpf_insn) + adjust_off;
+		if (insn_off > UINT32_MAX) {
+			pr_warn("invalid jump table value %llx at offset %d (adjust_off %d)\n",
+				jt[i], sym_off + i, adjust_off);
+			err = -EINVAL;
+			goto err_close;
+		}
+
+		val.orig_off = insn_off;
+		err = bpf_map_update_elem(map_fd, &i, &val, 0);
+		if (err)
+			goto err_close;
+	}
+
+	err = bpf_map_freeze(map_fd);
+	if (err)
+		goto err_close;
+
+	err = add_jt_map(obj, prog, sym_off, map_fd);
+	if (err)
+		goto err_close;
+
+	return map_fd;
+
+err_close:
+	close(map_fd);
+	return err;
+}
+
+/*
+ * In LLVM the .jumptables section contains jump tables entries relative to the
+ * section start. The BPF kernel-side code expects jump table offsets relative
+ * to the beginning of the program (passed in bpf(BPF_PROG_LOAD)). This helper
+ * computes a delta to be added when creating a map.
+ */
+static int jt_adjust_off(struct bpf_program *prog, int insn_idx)
+{
+	int i;
+
+	for (i = prog->subprog_cnt - 1; i >= 0; i--) {
+		if (insn_idx >= prog->subprogs[i].sub_insn_off)
+			return prog->subprogs[i].sub_insn_off - prog->subprogs[i].sec_insn_off;
+	}
+
+	return -prog->sec_insn_off;
+}
+
+
 /* Relocate data references within program code:
  *  - map references;
  *  - global variable references;
@@ -6235,6 +6422,21 @@ bpf_object__relocate_data(struct bpf_object *obj, struct bpf_program *prog)
 		case RELO_CORE:
 			/* will be handled by bpf_program_record_relos() */
 			break;
+		case RELO_INSN_ARRAY: {
+			int map_fd;
+
+			map_fd = create_jt_map(obj, prog, relo->sym_off, relo->sym_size,
+					       jt_adjust_off(prog, relo->insn_idx));
+			if (map_fd < 0) {
+				pr_warn("prog '%s': relo #%d: can't create jump table: sym_off %u\n",
+						prog->name, i, relo->sym_off);
+				return map_fd;
+			}
+			insn[0].src_reg = BPF_PSEUDO_MAP_VALUE;
+			insn->imm = map_fd;
+			insn->off = 0;
+		}
+			break;
 		default:
 			pr_warn("prog '%s': relo #%d: bad relo type %d\n",
 				prog->name, i, relo->type);
@@ -6432,6 +6634,24 @@ static int append_subprog_relos(struct bpf_program *main_prog, struct bpf_progra
 	return 0;
 }
 
+static int save_subprog_offsets(struct bpf_program *main_prog, struct bpf_program *subprog)
+{
+	size_t size = sizeof(main_prog->subprogs[0]);
+	int new_cnt = main_prog->subprog_cnt + 1;
+	void *tmp;
+
+	tmp = libbpf_reallocarray(main_prog->subprogs, new_cnt, size);
+	if (!tmp)
+		return -ENOMEM;
+
+	main_prog->subprogs = tmp;
+	main_prog->subprogs[new_cnt - 1].sec_insn_off = subprog->sec_insn_off;
+	main_prog->subprogs[new_cnt - 1].sub_insn_off = subprog->sub_insn_off;
+	main_prog->subprog_cnt = new_cnt;
+
+	return 0;
+}
+
 static int
 bpf_object__append_subprog_code(struct bpf_object *obj, struct bpf_program *main_prog,
 				struct bpf_program *subprog)
@@ -6461,6 +6681,15 @@ bpf_object__append_subprog_code(struct bpf_object *obj, struct bpf_program *main
 	err = append_subprog_relos(main_prog, subprog);
 	if (err)
 		return err;
+
+	/* Save subprogram offsets */
+	err = save_subprog_offsets(main_prog, subprog);
+	if (err) {
+		pr_warn("prog '%s': failed to add subprog offsets: %s\n",
+			main_prog->name, errstr(err));
+		return err;
+	}
+
 	return 0;
 }
 
@@ -9228,6 +9457,15 @@ void bpf_object__close(struct bpf_object *obj)
 
 	zfree(&obj->arena_data);
 
+	zfree(&obj->jumptables_data);
+	obj->jumptables_data_sz = 0;
+
+	if (obj->jumptable_maps && obj->jumptable_map_cnt) {
+		for (i = 0; i < obj->jumptable_map_cnt; i++)
+			close(obj->jumptable_maps[i].fd);
+	}
+	zfree(&obj->jumptable_maps);
+
 	free(obj);
 }
 
diff --git a/tools/lib/bpf/libbpf_probes.c b/tools/lib/bpf/libbpf_probes.c
index 9dfbe7750f56..bccf4bb747e1 100644
--- a/tools/lib/bpf/libbpf_probes.c
+++ b/tools/lib/bpf/libbpf_probes.c
@@ -364,6 +364,10 @@ static int probe_map_create(enum bpf_map_type map_type)
 	case BPF_MAP_TYPE_SOCKHASH:
 	case BPF_MAP_TYPE_REUSEPORT_SOCKARRAY:
 		break;
+	case BPF_MAP_TYPE_INSN_ARRAY:
+		key_size	= sizeof(__u32);
+		value_size	= sizeof(struct bpf_insn_array_value);
+		break;
 	case BPF_MAP_TYPE_UNSPEC:
 	default:
 		return -EOPNOTSUPP;
diff --git a/tools/lib/bpf/linker.c b/tools/lib/bpf/linker.c
index 56ae77047bc3..3defd4bc9154 100644
--- a/tools/lib/bpf/linker.c
+++ b/tools/lib/bpf/linker.c
@@ -27,6 +27,8 @@
 #include "strset.h"
 
 #define BTF_EXTERN_SEC ".extern"
+#define JUMPTABLES_SEC ".jumptables"
+#define JUMPTABLES_REL_SEC ".rel.jumptables"
 
 struct src_sec {
 	const char *sec_name;
@@ -2025,6 +2027,9 @@ static int linker_append_elf_sym(struct bpf_linker *linker, struct src_obj *obj,
 			obj->sym_map[src_sym_idx] = dst_sec->sec_sym_idx;
 			return 0;
 		}
+
+		if (strcmp(src_sec->sec_name, JUMPTABLES_SEC) == 0)
+			goto add_sym;
 	}
 
 	if (sym_bind == STB_LOCAL)
@@ -2271,8 +2276,9 @@ static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob
 						insn->imm += sec->dst_off / sizeof(struct bpf_insn);
 					else
 						insn->imm += sec->dst_off;
-				} else {
-					pr_warn("relocation against STT_SECTION in non-exec section is not supported!\n");
+				} else if (strcmp(src_sec->sec_name, JUMPTABLES_REL_SEC) != 0) {
+					pr_warn("relocation against STT_SECTION in section %s is not supported!\n",
+						src_sec->sec_name);
 					return -EINVAL;
 				}
 			}
-- 
2.34.1

[PATCH v6 bpf-next 15/17] bpftool: Recognize insn_array map type

[Anton Protopopov]

Teach bpftool to recognize instruction array map type.

Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
Acked-by: Quentin Monnet <qmo@kernel.org>
---
 tools/bpf/bpftool/Documentation/bpftool-map.rst | 3 ++-
 tools/bpf/bpftool/map.c                         | 3 ++-
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/tools/bpf/bpftool/Documentation/bpftool-map.rst b/tools/bpf/bpftool/Documentation/bpftool-map.rst
index 252e4c538edb..1af3305ea2b2 100644
--- a/tools/bpf/bpftool/Documentation/bpftool-map.rst
+++ b/tools/bpf/bpftool/Documentation/bpftool-map.rst
@@ -55,7 +55,8 @@ MAP COMMANDS
 |     | **devmap** | **devmap_hash** | **sockmap** | **cpumap** | **xskmap** | **sockhash**
 |     | **cgroup_storage** | **reuseport_sockarray** | **percpu_cgroup_storage**
 |     | **queue** | **stack** | **sk_storage** | **struct_ops** | **ringbuf** | **inode_storage**
-|     | **task_storage** | **bloom_filter** | **user_ringbuf** | **cgrp_storage** | **arena** }
+|     | **task_storage** | **bloom_filter** | **user_ringbuf** | **cgrp_storage** | **arena**
+|     | **insn_array** }
 
 DESCRIPTION
 ===========
diff --git a/tools/bpf/bpftool/map.c b/tools/bpf/bpftool/map.c
index c9de44a45778..7ebf7dbcfba4 100644
--- a/tools/bpf/bpftool/map.c
+++ b/tools/bpf/bpftool/map.c
@@ -1477,7 +1477,8 @@ static int do_help(int argc, char **argv)
 		"                 devmap | devmap_hash | sockmap | cpumap | xskmap | sockhash |\n"
 		"                 cgroup_storage | reuseport_sockarray | percpu_cgroup_storage |\n"
 		"                 queue | stack | sk_storage | struct_ops | ringbuf | inode_storage |\n"
-		"                 task_storage | bloom_filter | user_ringbuf | cgrp_storage | arena }\n"
+		"                 task_storage | bloom_filter | user_ringbuf | cgrp_storage | arena |\n"
+		"                 insn_array }\n"
 		"       " HELP_SPEC_OPTIONS " |\n"
 		"                    {-f|--bpffs} | {-n|--nomount} }\n"
 		"",
-- 
2.34.1

[PATCH v6 bpf-next 16/17] selftests/bpf: add new verifier_gotox test

[Anton Protopopov]

Add a set of tests to validate core gotox functionality
without need to rely on compilers.

Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
---
 .../selftests/bpf/prog_tests/verifier.c       |   2 +
 .../selftests/bpf/progs/verifier_gotox.c      | 277 ++++++++++++++++++
 2 files changed, 279 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_gotox.c

diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c
index c0e8ffdaa484..4b4b081b46cc 100644
--- a/tools/testing/selftests/bpf/prog_tests/verifier.c
+++ b/tools/testing/selftests/bpf/prog_tests/verifier.c
@@ -35,6 +35,7 @@
 #include "verifier_global_subprogs.skel.h"
 #include "verifier_global_ptr_args.skel.h"
 #include "verifier_gotol.skel.h"
+#include "verifier_gotox.skel.h"
 #include "verifier_helper_access_var_len.skel.h"
 #include "verifier_helper_packet_access.skel.h"
 #include "verifier_helper_restricted.skel.h"
@@ -173,6 +174,7 @@ void test_verifier_div_overflow(void)         { RUN(verifier_div_overflow); }
 void test_verifier_global_subprogs(void)      { RUN(verifier_global_subprogs); }
 void test_verifier_global_ptr_args(void)      { RUN(verifier_global_ptr_args); }
 void test_verifier_gotol(void)                { RUN(verifier_gotol); }
+void test_verifier_gotox(void)                { RUN(verifier_gotox); }
 void test_verifier_helper_access_var_len(void) { RUN(verifier_helper_access_var_len); }
 void test_verifier_helper_packet_access(void) { RUN(verifier_helper_packet_access); }
 void test_verifier_helper_restricted(void)    { RUN(verifier_helper_restricted); }
diff --git a/tools/testing/selftests/bpf/progs/verifier_gotox.c b/tools/testing/selftests/bpf/progs/verifier_gotox.c
new file mode 100644
index 000000000000..1a92e4d321e8
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/verifier_gotox.c
@@ -0,0 +1,277 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2025 Isovalent */
+
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+#include "../../../include/linux/filter.h"
+
+#ifdef __TARGET_ARCH_x86
+
+#define DEFINE_SIMPLE_JUMP_TABLE_PROG(NAME, SRC_REG, OFF, IMM, OUTCOME)	\
+									\
+	SEC("socket")							\
+	OUTCOME								\
+	__naked void jump_table_ ## NAME(void)				\
+	{								\
+		asm volatile ("						\
+		.pushsection .jumptables,\"\",@progbits;		\
+	jt0_%=:								\
+		.quad ret0_%=;						\
+		.quad ret1_%=;						\
+		.size jt0_%=, 16;					\
+		.global jt0_%=;						\
+		.popsection;						\
+									\
+		r0 = jt0_%= ll;						\
+		r0 += 8;						\
+		r0 = *(u64 *)(r0 + 0);					\
+		.8byte %[gotox_r0];					\
+		ret0_%=:						\
+		r0 = 0;							\
+		exit;							\
+		ret1_%=:						\
+		r0 = 1;							\
+		exit;							\
+	"	:							\
+		: __imm_insn(gotox_r0, BPF_RAW_INSN(BPF_JMP | BPF_JA | BPF_X, BPF_REG_0, (SRC_REG), (OFF) , (IMM))) \
+		: __clobber_all);					\
+	}
+
+/*
+ * The first program which doesn't use reserved fields
+ * loads and works properly. The rest fail to load.
+ */
+DEFINE_SIMPLE_JUMP_TABLE_PROG(ok,                          BPF_REG_0, 0, 0, __success __retval(1))
+DEFINE_SIMPLE_JUMP_TABLE_PROG(reserved_field_src_reg,      BPF_REG_1, 0, 0, __failure __msg("BPF_JA|BPF_X uses reserved fields"))
+DEFINE_SIMPLE_JUMP_TABLE_PROG(reserved_field_non_zero_off, BPF_REG_0, 1, 0, __failure __msg("BPF_JA|BPF_X uses reserved fields"))
+DEFINE_SIMPLE_JUMP_TABLE_PROG(reserved_field_non_zero_imm, BPF_REG_0, 0, 1, __failure __msg("BPF_JA|BPF_X uses reserved fields"))
+
+/*
+ * Gotox is forbidden when there is no jump table loaded
+ * which points to the sub-function where the gotox is used
+ */
+SEC("socket")
+__failure __msg("no jump tables found for subprog starting at 0")
+__naked void jump_table_no_jump_table(void)
+{
+	asm volatile ("						\
+	.8byte %[gotox_r0];					\
+	r0 = 1;							\
+	exit;							\
+"	:							\
+	: __imm_insn(gotox_r0, BPF_RAW_INSN(BPF_JMP | BPF_JA | BPF_X, BPF_REG_0, 0, 0 , 0))
+	: __clobber_all);
+}
+
+/*
+ * Incorrect type of the target register, only PTR_TO_INSN allowed
+ */
+SEC("socket")
+__failure __msg("R1 has type 1, expected PTR_TO_INSN")
+__naked void jump_table_incorrect_dst_reg_type(void)
+{
+	asm volatile ("						\
+	.pushsection .jumptables,\"\",@progbits;		\
+jt0_%=:								\
+	.quad ret0_%=;						\
+	.quad ret1_%=;						\
+	.size jt0_%=, 16;					\
+	.global jt0_%=;						\
+	.popsection;						\
+								\
+	r0 = jt0_%= ll;						\
+	r0 += 8;						\
+	r0 = *(u64 *)(r0 + 0);					\
+	r1 = 42;						\
+	.8byte %[gotox_r1];					\
+	ret0_%=:						\
+	r0 = 0;							\
+	exit;							\
+	ret1_%=:						\
+	r0 = 1;							\
+	exit;							\
+"	:							\
+	: __imm_insn(gotox_r1, BPF_RAW_INSN(BPF_JMP | BPF_JA | BPF_X, BPF_REG_1, 0, 0 , 0))
+	: __clobber_all);
+}
+
+#define DEFINE_INVALID_SIZE_PROG(READ_SIZE, OUTCOME)			\
+									\
+	SEC("socket")							\
+	OUTCOME								\
+	__naked void jump_table_invalid_read_size_ ## READ_SIZE(void)	\
+	{								\
+		asm volatile ("						\
+		.pushsection .jumptables,\"\",@progbits;		\
+	jt0_%=:								\
+		.quad ret0_%=;						\
+		.quad ret1_%=;						\
+		.size jt0_%=, 16;					\
+		.global jt0_%=;						\
+		.popsection;						\
+									\
+		r0 = jt0_%= ll;						\
+		r0 += 8;						\
+		r0 = *(" #READ_SIZE " *)(r0 + 0);			\
+		.8byte %[gotox_r0];					\
+		ret0_%=:						\
+		r0 = 0;							\
+		exit;							\
+		ret1_%=:						\
+		r0 = 1;							\
+		exit;							\
+	"	:							\
+		: __imm_insn(gotox_r0, BPF_RAW_INSN(BPF_JMP | BPF_JA | BPF_X, BPF_REG_0, 0, 0 , 0)) \
+		: __clobber_all);					\
+	}
+
+DEFINE_INVALID_SIZE_PROG(u32, __failure __msg("Invalid read of 4 bytes from insn_array"))
+DEFINE_INVALID_SIZE_PROG(u16, __failure __msg("Invalid read of 2 bytes from insn_array"))
+DEFINE_INVALID_SIZE_PROG(u8,  __failure __msg("Invalid read of 1 bytes from insn_array"))
+
+SEC("socket")
+__failure __msg("misaligned value access off 0+1+0 size 8")
+__naked void jump_table_misaligned_access(void)
+{
+	asm volatile ("						\
+	.pushsection .jumptables,\"\",@progbits;		\
+jt0_%=:								\
+	.quad ret0_%=;						\
+	.quad ret1_%=;						\
+	.size jt0_%=, 16;					\
+	.global jt0_%=;						\
+	.popsection;						\
+								\
+	r0 = jt0_%= ll;						\
+	r0 += 1;						\
+	r0 = *(u64 *)(r0 + 0);					\
+	.8byte %[gotox_r0];					\
+	ret0_%=:						\
+	r0 = 0;							\
+	exit;							\
+	ret1_%=:						\
+	r0 = 1;							\
+	exit;							\
+"	:							\
+	: __imm_insn(gotox_r0, BPF_RAW_INSN(BPF_JMP | BPF_JA | BPF_X, BPF_REG_0, 0, 0 , 0))
+	: __clobber_all);
+}
+
+SEC("socket")
+__failure __msg("invalid access to map value, value_size=16 off=24 size=8")
+__naked void jump_table_invalid_mem_acceess_pos(void)
+{
+	asm volatile ("						\
+	.pushsection .jumptables,\"\",@progbits;		\
+jt0_%=:								\
+	.quad ret0_%=;						\
+	.quad ret1_%=;						\
+	.size jt0_%=, 16;					\
+	.global jt0_%=;						\
+	.popsection;						\
+								\
+	r0 = jt0_%= ll;						\
+	r0 += 24;						\
+	r0 = *(u64 *)(r0 + 0);					\
+	.8byte %[gotox_r0];					\
+	ret0_%=:						\
+	r0 = 0;							\
+	exit;							\
+	ret1_%=:						\
+	r0 = 1;							\
+	exit;							\
+"	:							\
+	: __imm_insn(gotox_r0, BPF_RAW_INSN(BPF_JMP | BPF_JA | BPF_X, BPF_REG_0, 0, 0 , 0))
+	: __clobber_all);
+}
+
+SEC("socket")
+__failure __msg("invalid access to map value, value_size=16 off=-24 size=8")
+__naked void jump_table_invalid_mem_acceess_neg(void)
+{
+	asm volatile ("						\
+	.pushsection .jumptables,\"\",@progbits;		\
+jt0_%=:								\
+	.quad ret0_%=;						\
+	.quad ret1_%=;						\
+	.size jt0_%=, 16;					\
+	.global jt0_%=;						\
+	.popsection;						\
+								\
+	r0 = jt0_%= ll;						\
+	r0 -= 24;						\
+	r0 = *(u64 *)(r0 + 0);					\
+	.8byte %[gotox_r0];					\
+	ret0_%=:						\
+	r0 = 0;							\
+	exit;							\
+	ret1_%=:						\
+	r0 = 1;							\
+	exit;							\
+"	:							\
+	: __imm_insn(gotox_r0, BPF_RAW_INSN(BPF_JMP | BPF_JA | BPF_X, BPF_REG_0, 0, 0 , 0))
+	: __clobber_all);
+}
+
+SEC("socket")
+__success __retval(1)
+__naked void jump_table_add_sub_ok(void)
+{
+	asm volatile ("						\
+	.pushsection .jumptables,\"\",@progbits;		\
+jt0_%=:								\
+	.quad ret0_%=;						\
+	.quad ret1_%=;						\
+	.size jt0_%=, 16;					\
+	.global jt0_%=;						\
+	.popsection;						\
+								\
+	r0 = jt0_%= ll;						\
+	r0 -= 24;						\
+	r0 += 32;						\
+	r0 = *(u64 *)(r0 + 0);					\
+	.8byte %[gotox_r0];					\
+	ret0_%=:						\
+	r0 = 0;							\
+	exit;							\
+	ret1_%=:						\
+	r0 = 1;							\
+	exit;							\
+"	:							\
+	: __imm_insn(gotox_r0, BPF_RAW_INSN(BPF_JMP | BPF_JA | BPF_X, BPF_REG_0, 0, 0 , 0))
+	: __clobber_all);
+}
+
+SEC("socket")
+__failure __msg("writes into insn_array not allowed")
+__naked void jump_table_no_writes(void)
+{
+	asm volatile ("						\
+	.pushsection .jumptables,\"\",@progbits;		\
+jt0_%=:								\
+	.quad ret0_%=;						\
+	.quad ret1_%=;						\
+	.size jt0_%=, 16;					\
+	.global jt0_%=;						\
+	.popsection;						\
+								\
+	r0 = jt0_%= ll;						\
+	r0 += 8;						\
+	r1 = 0xbeef;						\
+	*(u64 *)(r0 + 0) = r1;					\
+	.8byte %[gotox_r0];					\
+	ret0_%=:						\
+	r0 = 0;							\
+	exit;							\
+	ret1_%=:						\
+	r0 = 1;							\
+	exit;							\
+"	:							\
+	: __imm_insn(gotox_r0, BPF_RAW_INSN(BPF_JMP | BPF_JA | BPF_X, BPF_REG_0, 0, 0 , 0))
+	: __clobber_all);
+}
+
+#endif /* __TARGET_ARCH_x86 */
+
+char _license[] SEC("license") = "GPL";
-- 
2.34.1

[PATCH v6 bpf-next 17/17] selftests/bpf: add C-level selftests for indirect jumps

[Anton Protopopov]

Add C-level selftests for indirect jumps to validate LLVM and libbpf
functionality. The tests are intentionally disabled, to be run
locally by developers, but will not make the CI red.

Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
---
 tools/testing/selftests/bpf/Makefile          |   4 +-
 .../selftests/bpf/prog_tests/bpf_gotox.c      | 185 ++++++++
 tools/testing/selftests/bpf/progs/bpf_gotox.c | 414 ++++++++++++++++++
 3 files changed, 602 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/bpf_gotox.c
 create mode 100644 tools/testing/selftests/bpf/progs/bpf_gotox.c

diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile
index 7437c325179e..a897cb31fe6d 100644
--- a/tools/testing/selftests/bpf/Makefile
+++ b/tools/testing/selftests/bpf/Makefile
@@ -454,7 +454,9 @@ BPF_CFLAGS = -g -Wall -Werror -D__TARGET_ARCH_$(SRCARCH) $(MENDIAN)	\
 	     -I$(abspath $(OUTPUT)/../usr/include)			\
 	     -std=gnu11		 					\
 	     -fno-strict-aliasing 					\
-	     -Wno-compare-distinct-pointer-types
+	     -Wno-compare-distinct-pointer-types			\
+	     -Wno-initializer-overrides					\
+	     #
 # TODO: enable me -Wsign-compare
 
 CLANG_CFLAGS = $(CLANG_SYS_INCLUDES)
diff --git a/tools/testing/selftests/bpf/prog_tests/bpf_gotox.c b/tools/testing/selftests/bpf/prog_tests/bpf_gotox.c
new file mode 100644
index 000000000000..4394654ac75a
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/bpf_gotox.c
@@ -0,0 +1,185 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <test_progs.h>
+
+#include <linux/if_ether.h>
+#include <linux/in.h>
+#include <linux/ip.h>
+#include <linux/ipv6.h>
+#include <linux/in6.h>
+#include <linux/udp.h>
+#include <linux/tcp.h>
+
+#include <sys/syscall.h>
+#include <bpf/bpf.h>
+
+#include "bpf_gotox.skel.h"
+
+/* Disable tests for now, as CI runs with LLVM-20 */
+#if 0
+static void __test_run(struct bpf_program *prog, void *ctx_in, size_t ctx_size_in)
+{
+	LIBBPF_OPTS(bpf_test_run_opts, topts,
+			    .ctx_in = ctx_in,
+			    .ctx_size_in = ctx_size_in,
+		   );
+	int err, prog_fd;
+
+	prog_fd = bpf_program__fd(prog);
+	err = bpf_prog_test_run_opts(prog_fd, &topts);
+	ASSERT_OK(err, "test_run_opts err");
+}
+
+static void check_simple(struct bpf_gotox *skel,
+			 struct bpf_program *prog,
+			 __u64 ctx_in,
+			 __u64 expected)
+{
+	skel->bss->ret_user = 0;
+
+	__test_run(prog, &ctx_in, sizeof(ctx_in));
+
+	if (!ASSERT_EQ(skel->bss->ret_user, expected, "skel->bss->ret_user"))
+		return;
+}
+
+static void check_simple_fentry(struct bpf_gotox *skel,
+				struct bpf_program *prog,
+				__u64 ctx_in,
+				__u64 expected)
+{
+	skel->bss->in_user = ctx_in;
+	skel->bss->ret_user = 0;
+
+	/* trigger */
+	usleep(1);
+
+	if (!ASSERT_EQ(skel->bss->ret_user, expected, "skel->bss->ret_user"))
+		return;
+}
+
+/* validate that for two loads of the same jump table libbpf generates only one map */
+static void check_one_map_two_jumps(struct bpf_gotox *skel)
+{
+	struct bpf_prog_info prog_info;
+	struct bpf_map_info map_info;
+	__u32 len;
+	__u32 map_ids[16];
+	int prog_fd, map_fd;
+	int ret;
+	int i;
+	bool seen = false;
+
+	memset(&prog_info, 0, sizeof(prog_info));
+	prog_info.map_ids = (long)map_ids;
+	prog_info.nr_map_ids = ARRAY_SIZE(map_ids);
+	prog_fd = bpf_program__fd(skel->progs.one_map_two_jumps);
+	if (!ASSERT_GE(prog_fd, 0, "bpf_program__fd(one_map_two_jumps)"))
+		return;
+
+	len = sizeof(prog_info);
+	ret = bpf_obj_get_info_by_fd(prog_fd, &prog_info, &len);
+	if (!ASSERT_OK(ret, "bpf_obj_get_info_by_fd(prog_fd)"))
+		return;
+
+	for (i = 0; i < prog_info.nr_map_ids; i++) {
+		map_fd  = bpf_map_get_fd_by_id(map_ids[i]);
+		if (!ASSERT_GE(map_fd, 0, "bpf_program__fd(one_map_two_jumps)"))
+			return;
+
+		len = sizeof(map_info);
+		memset(&map_info, 0, len);
+		ret = bpf_obj_get_info_by_fd(map_fd, &map_info, &len);
+		if (!ASSERT_OK(ret, "bpf_obj_get_info_by_fd(map_fd)")) {
+			close(map_fd);
+			return;
+		}
+
+		if (map_info.type == BPF_MAP_TYPE_INSN_ARRAY) {
+			if (!ASSERT_EQ(seen, false, "more than one INSN_ARRAY map")) {
+				close(map_fd);
+				return;
+			}
+			seen = true;
+		}
+		close(map_fd);
+	}
+
+	ASSERT_EQ(seen, true, "no INSN_ARRAY map");
+}
+
+static void check_gotox_skel(struct bpf_gotox *skel)
+{
+	int i;
+	__u64 in[]   = {0, 1, 2, 3, 4,  5, 77};
+	__u64 out[]  = {2, 3, 4, 5, 7, 19, 19};
+	__u64 out2[] = {103, 104, 107, 205, 115, 1019, 1019};
+	__u64 in3[]  = {0, 11, 27, 31, 22, 45, 99};
+	__u64 out3[] = {2,  3,  4,  5, 19, 19, 19};
+	__u64 in4[]  = {0, 1, 2, 3, 4,  5, 77};
+	__u64 out4[] = {12, 15, 7 , 15, 12, 15, 15};
+
+	for (i = 0; i < ARRAY_SIZE(in); i++)
+		check_simple(skel, skel->progs.simple_test, in[i], out[i]);
+
+	for (i = 0; i < ARRAY_SIZE(in); i++)
+		check_simple(skel, skel->progs.simple_test2, in[i], out[i]);
+
+	for (i = 0; i < ARRAY_SIZE(in); i++)
+		check_simple(skel, skel->progs.two_switches, in[i], out2[i]);
+
+	if (0) for (i = 0; i < ARRAY_SIZE(in); i++)
+		check_simple(skel, skel->progs.big_jump_table, in3[i], out3[i]);
+
+	if (0) for (i = 0; i < ARRAY_SIZE(in); i++)
+		check_simple(skel, skel->progs.one_jump_two_maps, in4[i], out4[i]);
+
+	for (i = 0; i < ARRAY_SIZE(in); i++)
+		check_simple(skel, skel->progs.use_static_global1, in[i], out[i]);
+
+	for (i = 0; i < ARRAY_SIZE(in); i++)
+		check_simple(skel, skel->progs.use_static_global2, in[i], out[i]);
+
+	for (i = 0; i < ARRAY_SIZE(in); i++)
+		check_simple(skel, skel->progs.use_nonstatic_global1, in[i], out[i]);
+
+	for (i = 0; i < ARRAY_SIZE(in); i++)
+		check_simple(skel, skel->progs.use_nonstatic_global2, in[i], out[i]);
+
+	bpf_program__attach(skel->progs.simple_test_other_sec);
+	for (i = 0; i < ARRAY_SIZE(in); i++)
+		check_simple_fentry(skel, skel->progs.simple_test_other_sec, in[i], out[i]);
+
+	bpf_program__attach(skel->progs.use_static_global_other_sec);
+	for (i = 0; i < ARRAY_SIZE(in); i++)
+		check_simple_fentry(skel, skel->progs.use_static_global_other_sec, in[i], out[i]);
+
+	bpf_program__attach(skel->progs.use_nonstatic_global_other_sec);
+	for (i = 0; i < ARRAY_SIZE(in); i++)
+		check_simple_fentry(skel, skel->progs.use_nonstatic_global_other_sec, in[i], out[i]);
+
+	if (0) check_one_map_two_jumps(skel);
+}
+
+void test_bpf_gotox(void)
+{
+	struct bpf_gotox *skel;
+	int ret;
+
+	skel = bpf_gotox__open();
+	if (!ASSERT_NEQ(skel, NULL, "bpf_gotox__open"))
+		return;
+
+	ret = bpf_gotox__load(skel);
+	if (!ASSERT_OK(ret, "bpf_gotox__load"))
+		return;
+
+	check_gotox_skel(skel);
+
+	bpf_gotox__destroy(skel);
+}
+#else
+void test_bpf_gotox(void)
+{
+}
+#endif
diff --git a/tools/testing/selftests/bpf/progs/bpf_gotox.c b/tools/testing/selftests/bpf/progs/bpf_gotox.c
new file mode 100644
index 000000000000..a867d2dbcf48
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/bpf_gotox.c
@@ -0,0 +1,414 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include "vmlinux.h"
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+#include <bpf/bpf_core_read.h>
+#include "bpf_misc.h"
+
+/* Disable tests for now, as CI runs with LLVM-20 */
+#if 0
+__u64 in_user;
+__u64 ret_user;
+
+struct simple_ctx {
+	__u64 x;
+};
+
+__u64 some_var;
+
+/*
+ * This function adds code which will be replaced by a different
+ * number of instructions by the verifier. This adds additional
+ * stress on testing the insn_array maps corresponding to indirect jumps.
+ */
+static __always_inline void adjust_insns(__u64 x)
+{
+	some_var ^= x + bpf_jiffies64();
+}
+
+SEC("syscall")
+int simple_test(struct simple_ctx *ctx)
+{
+	switch (ctx->x) {
+	case 0:
+		adjust_insns(ctx->x + 1);
+		ret_user = 2;
+		break;
+	case 1:
+		adjust_insns(ctx->x + 7);
+		ret_user = 3;
+		break;
+	case 2:
+		adjust_insns(ctx->x + 9);
+		ret_user = 4;
+		break;
+	case 3:
+		adjust_insns(ctx->x + 11);
+		ret_user = 5;
+		break;
+	case 4:
+		adjust_insns(ctx->x + 17);
+		ret_user = 7;
+		break;
+	default:
+		adjust_insns(ctx->x + 177);
+		ret_user = 19;
+		break;
+	}
+
+	return 0;
+}
+
+SEC("syscall")
+int simple_test2(struct simple_ctx *ctx)
+{
+	switch (ctx->x) {
+	case 0:
+		adjust_insns(ctx->x + 1);
+		ret_user = 2;
+		break;
+	case 1:
+		adjust_insns(ctx->x + 7);
+		ret_user = 3;
+		break;
+	case 2:
+		adjust_insns(ctx->x + 9);
+		ret_user = 4;
+		break;
+	case 3:
+		adjust_insns(ctx->x + 11);
+		ret_user = 5;
+		break;
+	case 4:
+		adjust_insns(ctx->x + 17);
+		ret_user = 7;
+		break;
+	default:
+		adjust_insns(ctx->x + 177);
+		ret_user = 19;
+		break;
+	}
+
+	return 0;
+}
+
+SEC("fentry/" SYS_PREFIX "sys_nanosleep")
+int simple_test_other_sec(struct pt_regs *ctx)
+{
+	__u64 x = in_user;
+
+	switch (x) {
+	case 0:
+		adjust_insns(x + 1);
+		ret_user = 2;
+		break;
+	case 1:
+		adjust_insns(x + 7);
+		ret_user = 3;
+		break;
+	case 2:
+		adjust_insns(x + 9);
+		ret_user = 4;
+		break;
+	case 3:
+		adjust_insns(x + 11);
+		ret_user = 5;
+		break;
+	case 4:
+		adjust_insns(x + 17);
+		ret_user = 7;
+		break;
+	default:
+		adjust_insns(x + 177);
+		ret_user = 19;
+		break;
+	}
+
+	return 0;
+}
+
+SEC("syscall")
+int two_switches(struct simple_ctx *ctx)
+{
+	switch (ctx->x) {
+	case 0:
+		adjust_insns(ctx->x + 1);
+		ret_user = 2;
+		break;
+	case 1:
+		adjust_insns(ctx->x + 7);
+		ret_user = 3;
+		break;
+	case 2:
+		adjust_insns(ctx->x + 9);
+		ret_user = 4;
+		break;
+	case 3:
+		adjust_insns(ctx->x + 11);
+		ret_user = 5;
+		break;
+	case 4:
+		adjust_insns(ctx->x + 17);
+		ret_user = 7;
+		break;
+	default:
+		adjust_insns(ctx->x + 177);
+		ret_user = 19;
+		break;
+	}
+
+	switch (ctx->x + !!ret_user) {
+	case 1:
+		adjust_insns(ctx->x + 7);
+		ret_user = 103;
+		break;
+	case 2:
+		adjust_insns(ctx->x + 9);
+		ret_user = 104;
+		break;
+	case 3:
+		adjust_insns(ctx->x + 11);
+		ret_user = 107;
+		break;
+	case 4:
+		adjust_insns(ctx->x + 11);
+		ret_user = 205;
+		break;
+	case 5:
+		adjust_insns(ctx->x + 11);
+		ret_user = 115;
+		break;
+	default:
+		adjust_insns(ctx->x + 177);
+		ret_user = 1019;
+		break;
+	}
+
+	return 0;
+}
+
+SEC("syscall")
+int big_jump_table(struct simple_ctx *ctx __attribute__((unused)))
+{
+#if 0
+	const void *const jt[256] = {
+		[0 ... 255] = &&default_label,
+		[0] = &&l0,
+		[11] = &&l11,
+		[27] = &&l27,
+		[31] = &&l31,
+	};
+
+	goto *jt[ctx->x & 0xff];
+
+l0:
+	adjust_insns(ctx->x + 1);
+	ret_user = 2;
+	return 0;
+
+l11:
+	adjust_insns(ctx->x + 7);
+	ret_user = 3;
+	return 0;
+
+l27:
+	adjust_insns(ctx->x + 9);
+	ret_user = 4;
+	return 0;
+
+l31:
+	adjust_insns(ctx->x + 11);
+	ret_user = 5;
+	return 0;
+
+default_label:
+	adjust_insns(ctx->x + 177);
+	ret_user = 19;
+	return 0;
+#else
+	return 0;
+#endif
+}
+
+SEC("syscall")
+int one_jump_two_maps(struct simple_ctx *ctx __attribute__((unused)))
+{
+#if 0
+	__label__ l1, l2, l3, l4;
+	void *jt1[2] = { &&l1, &&l2 };
+	void *jt2[2] = { &&l3, &&l4 };
+	unsigned int a = ctx->x % 2;
+	unsigned int b = (ctx->x / 2) % 2;
+	volatile int ret = 0;
+
+	if (!(a < 2 && b < 2))
+		return 19;
+
+	if (ctx->x % 2)
+		goto *jt1[a];
+	else
+		goto *jt2[b];
+
+	l1: ret += 1;
+	l2: ret += 3;
+	l3: ret += 5;
+	l4: ret += 7;
+
+	ret_user = ret;
+	return ret;
+#else
+	return 0;
+#endif
+}
+
+SEC("syscall")
+int one_map_two_jumps(struct simple_ctx *ctx __attribute__((unused)))
+{
+#if 0
+	__label__ l1, l2, l3;
+	void *jt[3] = { &&l1, &&l2, &&l3 };
+	unsigned int a = (ctx->x >> 2) & 1;
+	unsigned int b = (ctx->x >> 3) & 1;
+	volatile int ret = 0;
+
+	if (ctx->x % 2)
+		goto *jt[a];
+
+	if (ctx->x % 3)
+		goto *jt[a + b];
+
+	l1: ret += 3;
+	l2: ret += 5;
+	l3: ret += 7;
+
+	ret_user = ret;
+	return ret;
+#else
+	return 0;
+#endif
+}
+
+/* Just to introduce some non-zero offsets in .text */
+static __noinline int f0(volatile struct simple_ctx *ctx __arg_ctx)
+{
+	if (ctx)
+		return 1;
+	else
+		return 13;
+}
+
+SEC("syscall") int f1(struct simple_ctx *ctx)
+{
+	ret_user = 0;
+	return f0(ctx);
+}
+
+static __noinline int __static_global(__u64 x)
+{
+	switch (x) {
+	case 0:
+		adjust_insns(x + 1);
+		ret_user = 2;
+		break;
+	case 1:
+		adjust_insns(x + 7);
+		ret_user = 3;
+		break;
+	case 2:
+		adjust_insns(x + 9);
+		ret_user = 4;
+		break;
+	case 3:
+		adjust_insns(x + 11);
+		ret_user = 5;
+		break;
+	case 4:
+		adjust_insns(x + 17);
+		ret_user = 7;
+		break;
+	default:
+		adjust_insns(x + 177);
+		ret_user = 19;
+		break;
+	}
+
+	return 0;
+}
+
+SEC("syscall")
+int use_static_global1(struct simple_ctx *ctx)
+{
+	ret_user = 0;
+	return __static_global(ctx->x);
+}
+
+SEC("syscall")
+int use_static_global2(struct simple_ctx *ctx)
+{
+	ret_user = 0;
+	adjust_insns(ctx->x + 1);
+	return __static_global(ctx->x);
+}
+
+SEC("fentry/" SYS_PREFIX "sys_nanosleep")
+int use_static_global_other_sec(void *ctx)
+{
+	return __static_global(in_user);
+}
+
+__noinline int __nonstatic_global(__u64 x)
+{
+	switch (x) {
+	case 0:
+		adjust_insns(x + 1);
+		ret_user = 2;
+		break;
+	case 1:
+		adjust_insns(x + 7);
+		ret_user = 3;
+		break;
+	case 2:
+		adjust_insns(x + 9);
+		ret_user = 4;
+		break;
+	case 3:
+		adjust_insns(x + 11);
+		ret_user = 5;
+		break;
+	case 4:
+		adjust_insns(x + 17);
+		ret_user = 7;
+		break;
+	default:
+		adjust_insns(x + 177);
+		ret_user = 19;
+		break;
+	}
+
+	return 0;
+}
+
+SEC("syscall")
+int use_nonstatic_global1(struct simple_ctx *ctx)
+{
+	ret_user = 0;
+	return __nonstatic_global(ctx->x);
+}
+
+SEC("syscall")
+int use_nonstatic_global2(struct simple_ctx *ctx)
+{
+	ret_user = 0;
+	adjust_insns(ctx->x + 1);
+	return __nonstatic_global(ctx->x);
+}
+
+SEC("fentry/" SYS_PREFIX "sys_nanosleep")
+int use_nonstatic_global_other_sec(void *ctx)
+{
+	return __nonstatic_global(in_user);
+}
+#endif
+
+char _license[] SEC("license") = "GPL";
-- 
2.34.1

Re: [PATCH v6 bpf-next 10/17] bpf, x86: add support for indirect jumps

[Anton Protopopov]

On 25/10/19 08:21PM, Anton Protopopov wrote:
> Add support for a new instruction
> 
>     BPF_JMP|BPF_X|BPF_JA, SRC=0, DST=Rx, off=0, imm=0
>
> [...]
>
> +static struct bpf_iarray *
> +create_jt(int t, struct bpf_verifier_env *env, int fd)
> +{
> +	static struct bpf_subprog_info *subprog;
> +	int subprog_start, subprog_end;
> +	struct bpf_iarray *jt;
> +	int i;
> +
> +	subprog = bpf_find_containing_subprog(env, t);
> +	subprog_start = subprog->start;
> +	subprog_end = (subprog + 1)->start;
> +	jt = jt_from_subprog(env, subprog_start, subprog_end);
> +	if (IS_ERR(jt))
> +		return jt;
> +
> +	/* Check that the every element of the jump table fits within the given subprogram */
> +	for (i = 0; i < jt->cnt; i++) {
> +		if (jt->items[i] < subprog_start || jt->items[i] >= subprog_end) {
> +			verbose(env, "jump table for insn %d points outside of the subprog [%u,%u]",
> +					t, subprog_start, subprog_end);
> +			return ERR_PTR(-EINVAL);

AI found a bug here: jt should have been freed in this error path.
Will fix in the next version.

Re: [PATCH v6 bpf-next 08/17] bpf, x86: allow indirect jumps to r8...r15

[Anton Protopopov]

On 25/10/19 08:21PM, Anton Protopopov wrote:
> Currently the emit_indirect_jump() function only accepts one of the
> RAX, RCX, ..., RBP registers as the destination. Make it to accept
> R8, R9, ..., R15 as well, and make callers to pass BPF registers, not
> native registers. This is required to enable indirect jumps support
> in eBPF.
> 
> Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
> Acked-by: Eduard Zingerman <eddyz87@gmail.com>
> ---
>  arch/x86/net/bpf_jit_comp.c | 28 +++++++++++++++++++++-------
>  1 file changed, 21 insertions(+), 7 deletions(-)
> 
> diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
> index c8e628410d2c..7443465ce9a4 100644
> --- a/arch/x86/net/bpf_jit_comp.c
> +++ b/arch/x86/net/bpf_jit_comp.c
> @@ -660,24 +660,38 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type t,
>  
>  #define EMIT_LFENCE()	EMIT3(0x0F, 0xAE, 0xE8)
>  
> -static void emit_indirect_jump(u8 **pprog, int reg, u8 *ip)
> +static void __emit_indirect_jump(u8 **pprog, int reg, bool ereg)
>  {
>  	u8 *prog = *pprog;
>  
> +	if (ereg)
> +		EMIT1(0x41);
> +
> +	EMIT2(0xFF, 0xE0 + reg);
> +
> +	*pprog = prog;
> +}
> +
> +static void emit_indirect_jump(u8 **pprog, int bpf_reg, u8 *ip)
> +{
> +	u8 *prog = *pprog;
> +	int reg = reg2hex[bpf_reg];
> +	bool ereg = is_ereg(bpf_reg);
> +
>  	if (cpu_feature_enabled(X86_FEATURE_INDIRECT_THUNK_ITS)) {
>  		OPTIMIZER_HIDE_VAR(reg);
>  		emit_jump(&prog, its_static_thunk(reg), ip);

AI found bug here: its_static_thunk(reg) should use reg+8*ereg.
(The code here was not changed, however, before this patch this code
only was called for eax and ecx.) Will fix in the next version.

Also added verifier_gotox tests which validate that gotox works with
r0,...,r9.

> [...]

Re: [PATCH v6 bpf-next 04/17] bpf, x86: add new map type: instructions array

[Alexei Starovoitov]

On Sun, Oct 19, 2025 at 1:15 PM Anton Protopopov
<a.s.protopopov@gmail.com> wrote:
> diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
> index d4c93d9e73e4..c8e628410d2c 100644
> --- a/arch/x86/net/bpf_jit_comp.c
> +++ b/arch/x86/net/bpf_jit_comp.c
> @@ -1691,6 +1691,7 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image
>         prog = temp;
>
>         for (i = 1; i <= insn_cnt; i++, insn++) {
> +               u32 abs_xlated_off = bpf_prog->aux->subprog_start + i - 1;
>                 const s32 imm32 = insn->imm;
>                 u32 dst_reg = insn->dst_reg;
>                 u32 src_reg = insn->src_reg;
> @@ -2751,6 +2752,13 @@ st:                      if (is_imm8(insn->off))
>                                 return -EFAULT;
>                         }
>                         memcpy(rw_image + proglen, temp, ilen);
> +
> +                       /*
> +                        * Instruction arrays need to know how xlated code
> +                        * maps to jitted code
> +                        */
> +                       bpf_prog_update_insn_ptr(bpf_prog, abs_xlated_off, proglen,
> +                                                image + proglen);

...

> +void bpf_prog_update_insn_ptr(struct bpf_prog *prog,
> +                             u32 xlated_off,
> +                             u32 jitted_off,
> +                             void *jitted_ip)
> +{
> +       struct bpf_insn_array *insn_array;
> +       struct bpf_map *map;
> +       int i, j;
> +
> +       for (i = 0; i < prog->aux->used_map_cnt; i++) {
> +               map = prog->aux->used_maps[i];
> +               if (!is_insn_array(map))
> +                       continue;
> +
> +               insn_array = cast_insn_array(map);
> +               for (j = 0; j < map->max_entries; j++) {
> +                       if (insn_array->ptrs[j].user_value.xlated_off == xlated_off) {
> +                               insn_array->ips[j] = (long)jitted_ip;
> +                               insn_array->ptrs[j].jitted_ip = jitted_ip;
> +                               insn_array->ptrs[j].user_value.jitted_off = jitted_off;
> +                       }
> +               }
> +       }
> +}

This algorithm doesn't scale.
You're calling bpf_prog_update_insn_ptr() for every insn
and doing it as many times as they're JIT passes.
There could be up to 20 passes and hundreds of thousands of insns.
x86 JIT already keeps the mapping from insn to IP in jit_datat->addrs[].
Use it. Roughly like:
insn_array = cast_insn_array(map);
for (j = 0; j < map->max_entries; j++) {
   ip = addrs[insn_array->ptrs[j].user_value.xlated_off -
subprog_start] + image;
And this is done once per insn_array after JIT is done.

Re: [PATCH v6 bpf-next 00/17] BPF indirect jumps

[patchwork-bot+netdevbpf]

Hello:

This series was applied to bpf/bpf-next.git (master)
by Alexei Starovoitov <ast@kernel.org>:

On Sun, 19 Oct 2025 20:21:28 +0000 you wrote:
> This patchset implements a new type of map, instruction set, and uses
> it to build support for indirect branches in BPF (on x86). (The same
> map will be later used to provide support for indirect calls and static
> keys.) See [1], [2] for more context.
> 
> Short table of contents:
> 
> [...]

Here is the summary with links:
  - [v6,bpf-next,01/17] bpf: fix the return value of push_stack
    https://git.kernel.org/bpf/bpf-next/c/6ea5fc92a0fc
  - [v6,bpf-next,02/17] bpf: save the start of functions in bpf_prog_aux
    https://git.kernel.org/bpf/bpf-next/c/f7d72d0b3f43
  - [v6,bpf-next,03/17] bpf: generalize and export map_get_next_key for arrays
    https://git.kernel.org/bpf/bpf-next/c/44481e492532
  - [v6,bpf-next,04/17] bpf, x86: add new map type: instructions array
    (no matching commit)
  - [v6,bpf-next,05/17] selftests/bpf: add selftests for new insn_array map
    (no matching commit)
  - [v6,bpf-next,06/17] bpf: support instructions arrays with constants blinding
    (no matching commit)
  - [v6,bpf-next,07/17] selftests/bpf: test instructions arrays with blinding
    (no matching commit)
  - [v6,bpf-next,08/17] bpf, x86: allow indirect jumps to r8...r15
    (no matching commit)
  - [v6,bpf-next,09/17] bpf: make bpf_insn_successors to return a pointer
    https://git.kernel.org/bpf/bpf-next/c/2f69c5685427
  - [v6,bpf-next,10/17] bpf, x86: add support for indirect jumps
    (no matching commit)
  - [v6,bpf-next,11/17] bpf: disasm: add support for BPF_JMP|BPF_JA|BPF_X
    (no matching commit)
  - [v6,bpf-next,12/17] bpf, docs: do not state that indirect jumps are not supported
    (no matching commit)
  - [v6,bpf-next,13/17] libbpf: fix formatting of bpf_object__append_subprog_code
    https://git.kernel.org/bpf/bpf-next/c/e7586577b75f
  - [v6,bpf-next,14/17] libbpf: support llvm-generated indirect jumps
    (no matching commit)
  - [v6,bpf-next,15/17] bpftool: Recognize insn_array map type
    (no matching commit)
  - [v6,bpf-next,16/17] selftests/bpf: add new verifier_gotox test
    (no matching commit)
  - [v6,bpf-next,17/17] selftests/bpf: add C-level selftests for indirect jumps
    (no matching commit)

You are awesome, thank you!
-- 
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html

Re: [PATCH v6 bpf-next 04/17] bpf, x86: add new map type: instructions array

[Anton Protopopov]

On 25/10/21 10:49AM, Alexei Starovoitov wrote:
> On Sun, Oct 19, 2025 at 1:15 PM Anton Protopopov
> <a.s.protopopov@gmail.com> wrote:
> > diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
> > index d4c93d9e73e4..c8e628410d2c 100644
> > --- a/arch/x86/net/bpf_jit_comp.c
> > +++ b/arch/x86/net/bpf_jit_comp.c
> > @@ -1691,6 +1691,7 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image
> >         prog = temp;
> >
> >         for (i = 1; i <= insn_cnt; i++, insn++) {
> > +               u32 abs_xlated_off = bpf_prog->aux->subprog_start + i - 1;
> >                 const s32 imm32 = insn->imm;
> >                 u32 dst_reg = insn->dst_reg;
> >                 u32 src_reg = insn->src_reg;
> > @@ -2751,6 +2752,13 @@ st:                      if (is_imm8(insn->off))
> >                                 return -EFAULT;
> >                         }
> >                         memcpy(rw_image + proglen, temp, ilen);
> > +
> > +                       /*
> > +                        * Instruction arrays need to know how xlated code
> > +                        * maps to jitted code
> > +                        */
> > +                       bpf_prog_update_insn_ptr(bpf_prog, abs_xlated_off, proglen,
> > +                                                image + proglen);
> 
> ...
> 
> > +void bpf_prog_update_insn_ptr(struct bpf_prog *prog,
> > +                             u32 xlated_off,
> > +                             u32 jitted_off,
> > +                             void *jitted_ip)
> > +{
> > +       struct bpf_insn_array *insn_array;
> > +       struct bpf_map *map;
> > +       int i, j;
> > +
> > +       for (i = 0; i < prog->aux->used_map_cnt; i++) {
> > +               map = prog->aux->used_maps[i];
> > +               if (!is_insn_array(map))
> > +                       continue;
> > +
> > +               insn_array = cast_insn_array(map);
> > +               for (j = 0; j < map->max_entries; j++) {
> > +                       if (insn_array->ptrs[j].user_value.xlated_off == xlated_off) {
> > +                               insn_array->ips[j] = (long)jitted_ip;
> > +                               insn_array->ptrs[j].jitted_ip = jitted_ip;
> > +                               insn_array->ptrs[j].user_value.jitted_off = jitted_off;
> > +                       }
> > +               }
> > +       }
> > +}
> 
> This algorithm doesn't scale.
> You're calling bpf_prog_update_insn_ptr() for every insn
> and doing it as many times as they're JIT passes.
> There could be up to 20 passes and hundreds of thousands of insns.
> x86 JIT already keeps the mapping from insn to IP in jit_datat->addrs[].
> Use it. Roughly like:
> insn_array = cast_insn_array(map);
> for (j = 0; j < map->max_entries; j++) {
>    ip = addrs[insn_array->ptrs[j].user_value.xlated_off -
> subprog_start] + image;
> And this is done once per insn_array after JIT is done.

Thanks! Will do.

Re: [PATCH v6 bpf-next 12/17] bpf, docs: do not state that indirect jumps are not supported

[Eduard Zingerman]

On Sun, 2025-10-19 at 20:21 +0000, Anton Protopopov wrote:
> The linux-notes.rst states that indirect jump instruction "is not
> currently supported by the verifier". Remove this part as outdated.
> 
> Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
> ---
>  Documentation/bpf/linux-notes.rst | 8 --------
>  1 file changed, 8 deletions(-)
> 
> diff --git a/Documentation/bpf/linux-notes.rst b/Documentation/bpf/linux-notes.rst
> index 00d2693de025..64ac146a926f 100644
> --- a/Documentation/bpf/linux-notes.rst
> +++ b/Documentation/bpf/linux-notes.rst
> @@ -12,14 +12,6 @@ Byte swap instructions
>  
>  ``BPF_FROM_LE`` and ``BPF_FROM_BE`` exist as aliases for ``BPF_TO_LE`` and ``BPF_TO_BE`` respectively.
>  
> -Jump instructions
> -=================
> -
> -``BPF_CALL | BPF_X | BPF_JMP`` (0x8d), where the helper function
> -integer would be read from a specified register, is not currently supported
> -by the verifier.  Any programs with this instruction will fail to load
> -until such support is added.
> -
>  Maps
>  ====
>  

Nit: bpf/standardization/instruction-set.rst needs an update,
     we don't have anything about `JA|X|JMP` in the "Jump instructions"
     section there.

Re: [PATCH v6 bpf-next 12/17] bpf, docs: do not state that indirect jumps are not supported

[Anton Protopopov]

On 25/10/21 12:15PM, Eduard Zingerman wrote:
> On Sun, 2025-10-19 at 20:21 +0000, Anton Protopopov wrote:
> > The linux-notes.rst states that indirect jump instruction "is not
> > currently supported by the verifier". Remove this part as outdated.
> > 
> > Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
> > ---
> >  Documentation/bpf/linux-notes.rst | 8 --------
> >  1 file changed, 8 deletions(-)
> > 
> > diff --git a/Documentation/bpf/linux-notes.rst b/Documentation/bpf/linux-notes.rst
> > index 00d2693de025..64ac146a926f 100644
> > --- a/Documentation/bpf/linux-notes.rst
> > +++ b/Documentation/bpf/linux-notes.rst
> > @@ -12,14 +12,6 @@ Byte swap instructions
> >  
> >  ``BPF_FROM_LE`` and ``BPF_FROM_BE`` exist as aliases for ``BPF_TO_LE`` and ``BPF_TO_BE`` respectively.
> >  
> > -Jump instructions
> > -=================
> > -
> > -``BPF_CALL | BPF_X | BPF_JMP`` (0x8d), where the helper function
> > -integer would be read from a specified register, is not currently supported
> > -by the verifier.  Any programs with this instruction will fail to load
> > -until such support is added.
> > -
> >  Maps
> >  ====
> >  
> 
> Nit: bpf/standardization/instruction-set.rst needs an update,
>      we don't have anything about `JA|X|JMP` in the "Jump instructions"
>      section there.

Ah yes, thanks.

Also, there is a limitation listed in the llvm doc that -O0
can't be used due to absence of indirect jumps. I wonder if
there should be more limitations introduced since the doc
was written. (I've tried, briefly, to compile selftests with -O0,
but this fails for other reasons, and I didn't have time to dig
into this.)

Re: [PATCH v6 bpf-next 12/17] bpf, docs: do not state that indirect jumps are not supported

[Eduard Zingerman]

On Tue, 2025-10-21 at 19:32 +0000, Anton Protopopov wrote:
> On 25/10/21 12:15PM, Eduard Zingerman wrote:
> > On Sun, 2025-10-19 at 20:21 +0000, Anton Protopopov wrote:
> > > The linux-notes.rst states that indirect jump instruction "is not
> > > currently supported by the verifier". Remove this part as outdated.
> > > 
> > > Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
> > > ---
> > >  Documentation/bpf/linux-notes.rst | 8 --------
> > >  1 file changed, 8 deletions(-)
> > > 
> > > diff --git a/Documentation/bpf/linux-notes.rst b/Documentation/bpf/linux-notes.rst
> > > index 00d2693de025..64ac146a926f 100644
> > > --- a/Documentation/bpf/linux-notes.rst
> > > +++ b/Documentation/bpf/linux-notes.rst
> > > @@ -12,14 +12,6 @@ Byte swap instructions
> > >  
> > >  ``BPF_FROM_LE`` and ``BPF_FROM_BE`` exist as aliases for ``BPF_TO_LE`` and ``BPF_TO_BE`` respectively.
> > >  
> > > -Jump instructions
> > > -=================
> > > -
> > > -``BPF_CALL | BPF_X | BPF_JMP`` (0x8d), where the helper function
> > > -integer would be read from a specified register, is not currently supported
> > > -by the verifier.  Any programs with this instruction will fail to load
> > > -until such support is added.
> > > -
> > >  Maps
> > >  ====
> > >  
> > 
> > Nit: bpf/standardization/instruction-set.rst needs an update,
> >      we don't have anything about `JA|X|JMP` in the "Jump instructions"
> >      section there.
> 
> Ah yes, thanks.
> 
> Also, there is a limitation listed in the llvm doc that -O0
> can't be used due to absence of indirect jumps. I wonder if
> there should be more limitations introduced since the doc
> was written. (I've tried, briefly, to compile selftests with -O0,
> but this fails for other reasons, and I didn't have time to dig
> into this.)

Lets fill this section as we go.
From the top of my head, I can't say what will or will not happen to
verifier if O0 is used. Things that don't happen at O0 include:
- SROA (variables are always on stack);
- constant propagation;
- inlining;
- loop unrolling.

In theory, none of that should confuse verifier in its current state.
But I'm sure there are special cases.

Re: [PATCH v6 bpf-next 12/17] bpf, docs: do not state that indirect jumps are not supported

[Anton Protopopov]

On 25/10/21 12:36PM, Eduard Zingerman wrote:
> On Tue, 2025-10-21 at 19:32 +0000, Anton Protopopov wrote:
> > On 25/10/21 12:15PM, Eduard Zingerman wrote:
> > > On Sun, 2025-10-19 at 20:21 +0000, Anton Protopopov wrote:
> > > > The linux-notes.rst states that indirect jump instruction "is not
> > > > currently supported by the verifier". Remove this part as outdated.
> > > > 
> > > > Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
> > > > ---
> > > >  Documentation/bpf/linux-notes.rst | 8 --------
> > > >  1 file changed, 8 deletions(-)
> > > > 
> > > > diff --git a/Documentation/bpf/linux-notes.rst b/Documentation/bpf/linux-notes.rst
> > > > index 00d2693de025..64ac146a926f 100644
> > > > --- a/Documentation/bpf/linux-notes.rst
> > > > +++ b/Documentation/bpf/linux-notes.rst
> > > > @@ -12,14 +12,6 @@ Byte swap instructions
> > > >  
> > > >  ``BPF_FROM_LE`` and ``BPF_FROM_BE`` exist as aliases for ``BPF_TO_LE`` and ``BPF_TO_BE`` respectively.
> > > >  
> > > > -Jump instructions
> > > > -=================
> > > > -
> > > > -``BPF_CALL | BPF_X | BPF_JMP`` (0x8d), where the helper function
> > > > -integer would be read from a specified register, is not currently supported
> > > > -by the verifier.  Any programs with this instruction will fail to load
> > > > -until such support is added.
> > > > -
> > > >  Maps
> > > >  ====
> > > >  
> > > 
> > > Nit: bpf/standardization/instruction-set.rst needs an update,
> > >      we don't have anything about `JA|X|JMP` in the "Jump instructions"
> > >      section there.
> > 
> > Ah yes, thanks.
> > 
> > Also, there is a limitation listed in the llvm doc that -O0
> > can't be used due to absence of indirect jumps. I wonder if
> > there should be more limitations introduced since the doc
> > was written. (I've tried, briefly, to compile selftests with -O0,
> > but this fails for other reasons, and I didn't have time to dig
> > into this.)
> 
> Lets fill this section as we go.
> From the top of my head, I can't say what will or will not happen to
> verifier if O0 is used. Things that don't happen at O0 include:
> - SROA (variables are always on stack);
> - constant propagation;
> - inlining;
> - loop unrolling.
> 
> In theory, none of that should confuse verifier in its current state.
> But I'm sure there are special cases.

Stack was the first thing I've bumped into:

    progs/bpf_arena_spin_lock.h:164:12: error: Looks like the BPF stack limit is exceeded. Please move large on stack variables into BPF per-cpu array map. For non-kernel uses, the stack can be
          increased using -mllvm -bpf-stack-size.

      164 |         } while (!atomic_try_cmpxchg_relaxed(&lock->val, &old, new));
          |                   ^

But then also some things, say

    tools/testing/selftests/bpf/tools/include/bpf/bpf_helpers.h:169:15: error: invalid operand for inline asm constraint 'i'
      169 |         asm volatile("r1 = %[ctx]\n\t"
          |                      ^
                                 "r2 = %[map]\n\t"
                                 "r3 = %[slot]\n\t"
                                 "call 12"
                                 :: [ctx]"r"(ctx), [map]"r"(map), [slot]"i"(slot)
                                 : "r0", "r1", "r2", "r3", "r4", "r5");

Re: [PATCH v6 bpf-next 12/17] bpf, docs: do not state that indirect jumps are not supported

[Eduard Zingerman]

On Tue, 2025-10-21 at 19:50 +0000, Anton Protopopov wrote:

[...]

> Stack was the first thing I've bumped into:
> 
>     progs/bpf_arena_spin_lock.h:164:12: error: Looks like the BPF stack limit is exceeded. Please move large on stack variables into BPF per-cpu array map. For non-kernel uses, the stack can be
>           increased using -mllvm -bpf-stack-size.
> 
>       164 |         } while (!atomic_try_cmpxchg_relaxed(&lock->val, &old, new));
>           |                   ^

I mean, this makes sense ¯\_(ツ)_/¯.

> But then also some things, say
> 
>     tools/testing/selftests/bpf/tools/include/bpf/bpf_helpers.h:169:15: error: invalid operand for inline asm constraint 'i'
>       169 |         asm volatile("r1 = %[ctx]\n\t"
>           |                      ^
>                                  "r2 = %[map]\n\t"
>                                  "r3 = %[slot]\n\t"
>                                  "call 12"
>                                  :: [ctx]"r"(ctx), [map]"r"(map), [slot]"i"(slot)
>                                  : "r0", "r1", "r2", "r3", "r4", "r5");

That's probably because of the missing constant propagation.

Re: [PATCH v6 bpf-next 10/17] bpf, x86: add support for indirect jumps

[Eduard Zingerman]

On Sun, 2025-10-19 at 20:21 +0000, Anton Protopopov wrote:
> Add support for a new instruction
> 
>     BPF_JMP|BPF_X|BPF_JA, SRC=0, DST=Rx, off=0, imm=0
> 
> which does an indirect jump to a location stored in Rx.  The register
> Rx should have type PTR_TO_INSN. This new type assures that the Rx
> register contains a value (or a range of values) loaded from a
> correct jump table – map of type instruction array.
> 
> For example, for a C switch LLVM will generate the following code:
> 
>     0:   r3 = r1                    # "switch (r3)"
>     1:   if r3 > 0x13 goto +0x666   # check r3 boundaries
>     2:   r3 <<= 0x3                 # adjust to an index in array of addresses
>     3:   r1 = 0xbeef ll             # r1 is PTR_TO_MAP_VALUE, r1->map_ptr=M
>     5:   r1 += r3                   # r1 inherits boundaries from r3
>     6:   r1 = *(u64 *)(r1 + 0x0)    # r1 now has type INSN_TO_PTR
>     7:   gotox r1                   # jit will generate proper code
> 
> Here the gotox instruction corresponds to one particular map. This is
> possible however to have a gotox instruction which can be loaded from
> different maps, e.g.
> 
>     0:   r1 &= 0x1
>     1:   r2 <<= 0x3
>     2:   r3 = 0x0 ll                # load from map M_1
>     4:   r3 += r2
>     5:   if r1 == 0x0 goto +0x4
>     6:   r1 <<= 0x3
>     7:   r3 = 0x0 ll                # load from map M_2
>     9:   r3 += r1
>     A:   r1 = *(u64 *)(r3 + 0x0)
>     B:   gotox r1                   # jump to target loaded from M_1 or M_2
> 
> During check_cfg stage the verifier will collect all the maps which
> point to inside the subprog being verified. When building the config,
> the high 16 bytes of the insn_state are used, so this patch
> (theoretically) supports jump tables of up to 2^16 slots.
> 
> During the later stage, in check_indirect_jump, it is checked that
> the register Rx was loaded from a particular instruction array.
> 
> Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
> ---

LGTM, please, address a few remaining points.

Acked-by: Eduard Zingerman <eddyz87@gmail.com>

[...]

> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index ae017c032944..d2df21fde118 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c

[...]

> +static struct bpf_iarray *
> +create_jt(int t, struct bpf_verifier_env *env, int fd)
  		    	   		    	  ^^^^^^
				   This parameter is unused
> +{
> +	static struct bpf_subprog_info *subprog;
> +	int subprog_start, subprog_end;
> +	struct bpf_iarray *jt;
> +	int i;
> +
> +	subprog = bpf_find_containing_subprog(env, t);
> +	subprog_start = subprog->start;
> +	subprog_end = (subprog + 1)->start;
> +	jt = jt_from_subprog(env, subprog_start, subprog_end);
> +	if (IS_ERR(jt))
> +		return jt;
> +
> +	/* Check that the every element of the jump table fits within the given subprogram */
> +	for (i = 0; i < jt->cnt; i++) {
> +		if (jt->items[i] < subprog_start || jt->items[i] >= subprog_end) {
> +			verbose(env, "jump table for insn %d points outside of the subprog [%u,%u]",
> +					t, subprog_start, subprog_end);
> +			return ERR_PTR(-EINVAL);
> +		}
> +	}
> +
> +	return jt;
> +}

[...]

> +/* gotox *dst_reg */
> +static int check_indirect_jump(struct bpf_verifier_env *env, struct bpf_insn *insn)
> +{
> +	struct bpf_verifier_state *other_branch;
> +	struct bpf_reg_state *dst_reg;
> +	struct bpf_map *map;
> +	u32 min_index, max_index;
> +	int err = 0;
> +	int n;
> +	int i;
> +
> +	dst_reg = reg_state(env, insn->dst_reg);
> +	if (dst_reg->type != PTR_TO_INSN) {
> +		verbose(env, "R%d has type %d, expected PTR_TO_INSN\n",
> +			     insn->dst_reg, dst_reg->type);
> +		return -EINVAL;
> +	}
> +
> +	map = dst_reg->map_ptr;
> +	if (verifier_bug_if(!map, env, "R%d has an empty map pointer", insn->dst_reg))
> +		return -EFAULT;
> +
> +	if (verifier_bug_if(map->map_type != BPF_MAP_TYPE_INSN_ARRAY, env,
> +			    "R%d has incorrect map type %d", insn->dst_reg, map->map_type))
> +		return -EFAULT;

Nit: we discussed this in v5, let's drop the verifier_bug_if() and
     return -EINVAL?

     > The program can be written in a way, such that e.g. hash map
     > pointer is passed as a parameter for gotox, that would be an
     > incorrect program, not a verifier bug.

     Also, use reg_type_str() instead of "type %d"?

> +
> +	err = indirect_jump_min_max_index(env, insn->dst_reg, map, &min_index, &max_index);
> +	if (err)
> +		return err;
> +
> +	/* Ensure that the buffer is large enough */
> +	if (!env->gotox_tmp_buf || env->gotox_tmp_buf->cnt < max_index - min_index + 1) {
> +		env->gotox_tmp_buf = iarray_realloc(env->gotox_tmp_buf,
> +						    max_index - min_index + 1);
> +		if (!env->gotox_tmp_buf)
> +			return -ENOMEM;
> +	}
> +
> +	n = copy_insn_array_uniq(map, min_index, max_index, env->gotox_tmp_buf->items);

Nit: let's not forget about a follow-up to remove this allocation.

> +	if (n < 0)
> +		return n;
> +	if (n == 0) {
> +		verbose(env, "register R%d doesn't point to any offset in map id=%d\n",
> +			     insn->dst_reg, map->id);
> +		return -EINVAL;
> +	}
> +
> +	for (i = 0; i < n - 1; i++) {
> +		other_branch = push_stack(env, env->gotox_tmp_buf->items[i],
> +					  env->insn_idx, env->cur_state->speculative);
> +		if (IS_ERR(other_branch))
> +			return PTR_ERR(other_branch);
> +	}
> +	env->insn_idx = env->gotox_tmp_buf->items[n-1];
> +	return 0;
> +}
> +

[...]

Re: [PATCH v6 bpf-next 11/17] bpf: disasm: add support for BPF_JMP|BPF_JA|BPF_X

[Eduard Zingerman]

On Sun, 2025-10-19 at 20:21 +0000, Anton Protopopov wrote:
> Add support for indirect jump instruction.
> 
> Example output from bpftool:
> 
>    0: (79) r3 = *(u64 *)(r1 +0)
>    1: (25) if r3 > 0x4 goto pc+666
>    2: (67) r3 <<= 3
>    3: (18) r1 = 0xffffbeefspameggs
>    5: (0f) r1 += r3
>    6: (79) r1 = *(u64 *)(r1 +0)
>    7: (0d) gotox r1
> 
> Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
> ---

Acked-by: Eduard Zingerman <eddyz87@gmail.com>

[...]

Re: [PATCH v6 bpf-next 14/17] libbpf: support llvm-generated indirect jumps

[Eduard Zingerman]

On Sun, 2025-10-19 at 20:21 +0000, Anton Protopopov wrote:

[...]

> ---
>  tools/lib/bpf/libbpf.c        | 240 +++++++++++++++++++++++++++++++++-
>  tools/lib/bpf/libbpf_probes.c |   4 +
>  tools/lib/bpf/linker.c        |  10 +-
>  3 files changed, 251 insertions(+), 3 deletions(-)
> 
> diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
> index b90574f39d1c..ee44bc49a3ba 100644
> --- a/tools/lib/bpf/libbpf.c
> +++ b/tools/lib/bpf/libbpf.c

[...]

> +/*
> + * In LLVM the .jumptables section contains jump tables entries relative to the
> + * section start. The BPF kernel-side code expects jump table offsets relative
> + * to the beginning of the program (passed in bpf(BPF_PROG_LOAD)). This helper
> + * computes a delta to be added when creating a map.
> + */
> +static int jt_adjust_off(struct bpf_program *prog, int insn_idx)
> +{
> +	int i;
> +
> +	for (i = prog->subprog_cnt - 1; i >= 0; i--) {
> +		if (insn_idx >= prog->subprogs[i].sub_insn_off)

Sorry, I'm still confused about what happens here.
The `insn_idx` is comes from relocation, meaning that it is a value
recorded relative to section start, right?  On the other hand,
`.sub_insn_off` is an offset of a subprogram within a concatenated
program, about to be loaded.  These values should not be compared
directly.

I think, that my suggestion from v5 [1] should be easier to understand:

   > Or rename this thing to find_subprog_idx(), pass relo object into
   > create_jt_map(), call find_subprog_idx() there, and do the following:
   >
   >   xlated_off = jt[i] / sizeof(struct bpf_insn);
   >   /* make xlated_off relative to subprogram start */
   >   xlated_off -= prog->subprogs[subprog_idx].sec_insn_off;
   >   /* make xlated_off relative to main subprogram start */
   >   xlated_off += prog->subprogs[subprog_idx].sub_insn_off;

[1] https://lore.kernel.org/bpf/b5fd31c3e703c8c84c6710f5536510fbce04b36f.camel@gmail.com/

> +			return prog->subprogs[i].sub_insn_off - prog->subprogs[i].sec_insn_off;
> +	}
> +
> +	return -prog->sec_insn_off;
> +}
> +
> +
>  /* Relocate data references within program code:
>   *  - map references;
>   *  - global variable references;
> @@ -6235,6 +6422,21 @@ bpf_object__relocate_data(struct bpf_object *obj, struct bpf_program *prog)
>  		case RELO_CORE:
>  			/* will be handled by bpf_program_record_relos() */
>  			break;
> +		case RELO_INSN_ARRAY: {
> +			int map_fd;
> +
> +			map_fd = create_jt_map(obj, prog, relo->sym_off, relo->sym_size,
> +					       jt_adjust_off(prog, relo->insn_idx));
> +			if (map_fd < 0) {
> +				pr_warn("prog '%s': relo #%d: can't create jump table: sym_off %u\n",
> +						prog->name, i, relo->sym_off);
> +				return map_fd;
> +			}
> +			insn[0].src_reg = BPF_PSEUDO_MAP_VALUE;
> +			insn->imm = map_fd;
> +			insn->off = 0;
> +		}
> +			break;
>  		default:
>  			pr_warn("prog '%s': relo #%d: bad relo type %d\n",
>  				prog->name, i, relo->type);

[...]

> @@ -9228,6 +9457,15 @@ void bpf_object__close(struct bpf_object *obj)
>  
>  	zfree(&obj->arena_data);
>  
> +	zfree(&obj->jumptables_data);
> +	obj->jumptables_data_sz = 0;
> +
> +	if (obj->jumptable_maps && obj->jumptable_map_cnt) {

Nit: outer 'if' seems unnecessary.

> +		for (i = 0; i < obj->jumptable_map_cnt; i++)
> +			close(obj->jumptable_maps[i].fd);
> +	}
> +	zfree(&obj->jumptable_maps);
> +
>  	free(obj);
>  }

[...]

> diff --git a/tools/lib/bpf/linker.c b/tools/lib/bpf/linker.c
> index 56ae77047bc3..3defd4bc9154 100644
> --- a/tools/lib/bpf/linker.c
> +++ b/tools/lib/bpf/linker.c
> @@ -27,6 +27,8 @@
>  #include "strset.h"
>  
>  #define BTF_EXTERN_SEC ".extern"
> +#define JUMPTABLES_SEC ".jumptables"
> +#define JUMPTABLES_REL_SEC ".rel.jumptables"

Nit: maybe avoid duplicating JUMPTABLES_SEC by moving all *_SEC macro
     to libbpf_internal.h?

>  
>  struct src_sec {
>  	const char *sec_name;
> @@ -2025,6 +2027,9 @@ static int linker_append_elf_sym(struct bpf_linker *linker, struct src_obj *obj,
>  			obj->sym_map[src_sym_idx] = dst_sec->sec_sym_idx;
>  			return 0;
>  		}
> +
> +		if (strcmp(src_sec->sec_name, JUMPTABLES_SEC) == 0)
> +			goto add_sym;
>  	}
>  
>  	if (sym_bind == STB_LOCAL)
> @@ -2271,8 +2276,9 @@ static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob
>  						insn->imm += sec->dst_off / sizeof(struct bpf_insn);
>  					else
>  						insn->imm += sec->dst_off;
> -				} else {
> -					pr_warn("relocation against STT_SECTION in non-exec section is not supported!\n");
> +				} else if (strcmp(src_sec->sec_name, JUMPTABLES_REL_SEC) != 0) {
> +					pr_warn("relocation against STT_SECTION in section %s is not supported!\n",
> +						src_sec->sec_name);

Sorry, I missed this on a previous iteration.
LLVM generates section relative offsets for jump table contents, so it
seems that relocations inside jump table section should not occur.
Is this a leftover, or am I confused?

>  					return -EINVAL;
>  				}
>  			}

Re: [PATCH v6 bpf-next 16/17] selftests/bpf: add new verifier_gotox test

[Eduard Zingerman]

On Sun, 2025-10-19 at 20:21 +0000, Anton Protopopov wrote:
> Add a set of tests to validate core gotox functionality
> without need to rely on compilers.
> 
> Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
> ---

Thank you for adding these.
Could you please also add a test cases that checks the following errors:
- "jump table for insn %d points outside of the subprog [%u,%u]"
- "the sum of R%u umin_value %llu and off %u is too big\n"
- "register R%d doesn't point to any offset in map id=%d\n"
?

Might be the case that some of these can't be triggered because of the
check_mem_access() call.

[...]

> diff --git a/tools/testing/selftests/bpf/progs/verifier_gotox.c b/tools/testing/selftests/bpf/progs/verifier_gotox.c
> new file mode 100644
> index 000000000000..1a92e4d321e8
> --- /dev/null
> +++ b/tools/testing/selftests/bpf/progs/verifier_gotox.c

[...]

> +/*
> + * Gotox is forbidden when there is no jump table loaded
> + * which points to the sub-function where the gotox is used
> + */
> +SEC("socket")
> +__failure __msg("no jump tables found for subprog starting at 0")
                                                              ^^^^
				Nit: one day we need to figure out a way to
				     report subprogram names, when reporting
				     check_cfg() errors.

> +__naked void jump_table_no_jump_table(void)
> +{
> +	asm volatile ("						\
> +	.8byte %[gotox_r0];					\
> +	r0 = 1;							\
> +	exit;							\
> +"	:							\
> +	: __imm_insn(gotox_r0, BPF_RAW_INSN(BPF_JMP | BPF_JA | BPF_X, BPF_REG_0, 0, 0 , 0))
> +	: __clobber_all);
> +}
> +
> +/*
> + * Incorrect type of the target register, only PTR_TO_INSN allowed
> + */
> +SEC("socket")
> +__failure __msg("R1 has type 1, expected PTR_TO_INSN")
                           ^^^^^^
	      log.c:reg_type_str() should help here.

> +__naked void jump_table_incorrect_dst_reg_type(void)
> +{
> +	asm volatile ("						\
> +	.pushsection .jumptables,\"\",@progbits;		\
> +jt0_%=:								\
> +	.quad ret0_%=;						\
> +	.quad ret1_%=;						\
> +	.size jt0_%=, 16;					\
> +	.global jt0_%=;						\
> +	.popsection;						\
> +								\
> +	r0 = jt0_%= ll;						\
> +	r0 += 8;						\
> +	r0 = *(u64 *)(r0 + 0);					\
> +	r1 = 42;						\
> +	.8byte %[gotox_r1];					\
> +	ret0_%=:						\
> +	r0 = 0;							\
> +	exit;							\
> +	ret1_%=:						\
> +	r0 = 1;							\
> +	exit;							\
> +"	:							\
> +	: __imm_insn(gotox_r1, BPF_RAW_INSN(BPF_JMP | BPF_JA | BPF_X, BPF_REG_1, 0, 0 , 0))
> +	: __clobber_all);
> +}
> +
> +#define DEFINE_INVALID_SIZE_PROG(READ_SIZE, OUTCOME)			\

Nit: this can be merged with DEFINE_SIMPLE_JUMP_TABLE_PROG.

> +									\
> +	SEC("socket")							\
> +	OUTCOME								\
> +	__naked void jump_table_invalid_read_size_ ## READ_SIZE(void)	\
> +	{								\
> +		asm volatile ("						\
> +		.pushsection .jumptables,\"\",@progbits;		\
> +	jt0_%=:								\
> +		.quad ret0_%=;						\
> +		.quad ret1_%=;						\
> +		.size jt0_%=, 16;					\
> +		.global jt0_%=;						\
> +		.popsection;						\
> +									\
> +		r0 = jt0_%= ll;						\
> +		r0 += 8;						\
> +		r0 = *(" #READ_SIZE " *)(r0 + 0);			\
> +		.8byte %[gotox_r0];					\
> +		ret0_%=:						\
> +		r0 = 0;							\
> +		exit;							\
> +		ret1_%=:						\
> +		r0 = 1;							\
> +		exit;							\
> +	"	:							\
> +		: __imm_insn(gotox_r0, BPF_RAW_INSN(BPF_JMP | BPF_JA | BPF_X, BPF_REG_0, 0, 0 , 0)) \
> +		: __clobber_all);					\
> +	}
> +
> +DEFINE_INVALID_SIZE_PROG(u32, __failure __msg("Invalid read of 4 bytes from insn_array"))
> +DEFINE_INVALID_SIZE_PROG(u16, __failure __msg("Invalid read of 2 bytes from insn_array"))
> +DEFINE_INVALID_SIZE_PROG(u8,  __failure __msg("Invalid read of 1 bytes from insn_array"))

[...]

Re: [PATCH v6 bpf-next 14/17] libbpf: support llvm-generated indirect jumps

[Eduard Zingerman]

On Tue, 2025-10-21 at 15:18 -0700, Eduard Zingerman wrote:

[...]

> > +/*
> > + * In LLVM the .jumptables section contains jump tables entries relative to the
> > + * section start. The BPF kernel-side code expects jump table offsets relative
> > + * to the beginning of the program (passed in bpf(BPF_PROG_LOAD)). This helper
> > + * computes a delta to be added when creating a map.
> > + */
> > +static int jt_adjust_off(struct bpf_program *prog, int insn_idx)
> > +{
> > +	int i;
> > +
> > +	for (i = prog->subprog_cnt - 1; i >= 0; i--) {
> > +		if (insn_idx >= prog->subprogs[i].sub_insn_off)
> 
> Sorry, I'm still confused about what happens here.
> The `insn_idx` is comes from relocation, meaning that it is a value
> recorded relative to section start, right?  On the other hand,
> `.sub_insn_off` is an offset of a subprogram within a concatenated
> program, about to be loaded.  These values should not be compared
> directly.

I'm wrong on this account, append_subprog_relos() adjusts relo->insn_idx.
Still, please consider refactoring as below.

> I think, that my suggestion from v5 [1] should be easier to understand:
> 
>    > Or rename this thing to find_subprog_idx(), pass relo object into
>    > create_jt_map(), call find_subprog_idx() there, and do the following:
>    >
>    >   xlated_off = jt[i] / sizeof(struct bpf_insn);
>    >   /* make xlated_off relative to subprogram start */
>    >   xlated_off -= prog->subprogs[subprog_idx].sec_insn_off;
>    >   /* make xlated_off relative to main subprogram start */
>    >   xlated_off += prog->subprogs[subprog_idx].sub_insn_off;
> 
> [1] https://lore.kernel.org/bpf/b5fd31c3e703c8c84c6710f5536510fbce04b36f.camel@gmail.com/

[...]

Re: [PATCH v6 bpf-next 04/17] bpf, x86: add new map type: instructions array

[Eduard Zingerman]

On Sun, 2025-10-19 at 20:21 +0000, Anton Protopopov wrote:

[...]

> The functionality provided by this patch will be extended in consequent
> patches to implement BPF Static Keys, indirect jumps, and indirect calls.
> 
> Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
> ---

Aside from what Alexei pointed out, I only have a nit regarding jitted_ip.

Reviewed-by: Eduard Zingerman <eddyz87@gmail.com>

[...]

> diff --git a/include/linux/bpf.h b/include/linux/bpf.h
> index e53cda0aabb6..363355628d2e 100644
> --- a/include/linux/bpf.h
> +++ b/include/linux/bpf.h
> @@ -3789,4 +3789,40 @@ int bpf_prog_get_file_line(struct bpf_prog *prog, unsigned long ip, const char *
>  			   const char **linep, int *nump);

[...]

> +/*
> + * The struct bpf_insn_ptr structure describes a pointer to a
> + * particular instruction in a loaded BPF program. Initially
> + * it is initialised from userspace via user_value.xlated_off.
> + * During the program verification all other fields are populated
> + * accordingly:
> + *
> + *   jitted_ip:       address of the instruction in the jitted image
> + *   user_value:      user-visible original, xlated, and jitted offsets
> + */
> +struct bpf_insn_ptr {
> +	void *jitted_ip;

I think this one is no longer used anywhere.
I see it set in bpf_prog_update_insn_ptr() but it is not read anywhere.

> +	struct bpf_insn_array_value user_value;
> +};
> +

[...]

> diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
> index 6829936d33f5..805d441363cd 100644

[...]

> @@ -7645,4 +7646,24 @@ enum bpf_kfunc_flags {
>  	BPF_F_PAD_ZEROS = (1ULL << 0),
>  };
>  
> +/*
> + * Values of a BPF_MAP_TYPE_INSN_ARRAY entry must be of this type.
> + *
> + * Before the map is used the orig_off field should point to an
> + * instruction inside the program being loaded. The other fields
> + * must be set to 0.
> + *
> + * After the program is loaded, the xlated_off will be adjusted
> + * by the verifier to point to the index of the original instruction
> + * in the xlated program. If the instruction is deleted, it will
> + * be set to (u32)-1. The jitted_off will be set to the corresponding
> + * offset in the jitted image of the program.
> + */
> +struct bpf_insn_array_value {
> +	__u32 orig_off;
> +	__u32 xlated_off;
> +	__u32 jitted_off;
> +	__u32 :32;

This :u32, is it for alignment or for future extensibility?

> +};
> +

[...]

> diff --git a/kernel/bpf/bpf_insn_array.c b/kernel/bpf/bpf_insn_array.c
> new file mode 100644

[...]

> @@ -0,0 +1,288 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +/* Copyright (c) 2025 Isovalent */
> +
> +#include <linux/bpf.h>
> +
> +#define MAX_INSN_ARRAY_ENTRIES 256
> +
> +struct bpf_insn_array {
> +	struct bpf_map map;
> +	atomic_t used;
> +	long *ips;
> +	DECLARE_FLEX_ARRAY(struct bpf_insn_ptr, ptrs);
> +};

[...]

> +static inline u32 insn_array_alloc_size(u32 max_entries)
> +{
> +	const u32 base_size = sizeof(struct bpf_insn_array);
> +	const u32 entry_size = sizeof(struct bpf_insn_ptr);
> +
> +	return base_size + entry_size * max_entries;
> +}

Since this is doing a flexible array thing anyway, maybe also include
size for `ips` here? And in insn_array_alloc() point it at the tail
area.

[...]

Re: [PATCH v6 bpf-next 05/17] selftests/bpf: add selftests for new insn_array map

[Eduard Zingerman]

On Sun, 2025-10-19 at 20:21 +0000, Anton Protopopov wrote:
> Add the following selftests for new insn_array map:
> 
>   * Incorrect instruction indexes are rejected
>   * Two programs can't use the same map
>   * BPF progs can't operate the map
>   * no changes to code => map is the same
>   * expected changes when instructions are added
>   * expected changes when instructions are deleted
>   * expected changes when multiple functions are present
> 
> Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
> ---

Acked-by: Eduard Zingerman <eddyz87@gmail.com>

>  .../selftests/bpf/prog_tests/bpf_insn_array.c | 404 ++++++++++++++++++
>  1 file changed, 404 insertions(+)
>  create mode 100644 tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c
> 
> diff --git a/tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c b/tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c
> new file mode 100644
> index 000000000000..a4304ef5be13
> --- /dev/null
> +++ b/tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c

[...]

> +static void check_bpf_no_lookup(void)

This one can be moved to prog_tests/bpf_insn_array.c, I think.

> +{
> +	struct bpf_insn insns[] = {
> +		BPF_LD_MAP_FD(BPF_REG_1, 0),
> +		BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0),
> +		BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
> +		BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8),
> +		BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem),
> +		BPF_EXIT_INSN(),
> +	};
> +	int prog_fd = -1, map_fd;
> +
> +	map_fd = map_create(BPF_MAP_TYPE_INSN_ARRAY, 1);
> +	if (!ASSERT_GE(map_fd, 0, "map_create"))
> +		return;
> +
> +	insns[0].imm = map_fd;
> +
> +	if (!ASSERT_EQ(bpf_map_freeze(map_fd), 0, "bpf_map_freeze"))
> +		goto cleanup;
> +
> +	prog_fd = prog_load(insns, ARRAY_SIZE(insns), NULL, 0);
> +	if (!ASSERT_EQ(prog_fd, -EINVAL, "program should have been rejected (prog_fd != -EINVAL)"))
> +		goto cleanup;
> +
> +	/* correctness: check that prog is still loadable with normal map */
> +	close(map_fd);
> +	map_fd = map_create(BPF_MAP_TYPE_ARRAY, 1);
> +	insns[0].imm = map_fd;
> +	prog_fd = prog_load(insns, ARRAY_SIZE(insns), NULL, 0);
> +	if (!ASSERT_GE(prog_fd, 0, "bpf(BPF_PROG_LOAD)"))
> +		goto cleanup;
> +
> +cleanup:
> +	close(prog_fd);
> +	close(map_fd);
> +}

[...]

Re: [PATCH v6 bpf-next 17/17] selftests/bpf: add C-level selftests for indirect jumps

[Eduard Zingerman]

On Sun, 2025-10-19 at 20:21 +0000, Anton Protopopov wrote:

[...]

> diff --git a/tools/testing/selftests/bpf/prog_tests/bpf_gotox.c b/tools/testing/selftests/bpf/prog_tests/bpf_gotox.c
> new file mode 100644
> index 000000000000..4394654ac75a
> --- /dev/null
> +++ b/tools/testing/selftests/bpf/prog_tests/bpf_gotox.c
> @@ -0,0 +1,185 @@
> +// SPDX-License-Identifier: GPL-2.0
> +
> +#include <test_progs.h>
> +
> +#include <linux/if_ether.h>
> +#include <linux/in.h>
> +#include <linux/ip.h>
> +#include <linux/ipv6.h>
> +#include <linux/in6.h>
> +#include <linux/udp.h>
> +#include <linux/tcp.h>
> +
> +#include <sys/syscall.h>
> +#include <bpf/bpf.h>
> +
> +#include "bpf_gotox.skel.h"
> +
> +/* Disable tests for now, as CI runs with LLVM-20 */
> +#if 0

I removed all the `#if 0` and tried to compile this test using LLVM at commit
1a9aba29b09e ("[RISCV] Remove unreachable break statements. NFC (#164481)").

LLVM refuses to compile showing errors like below:

  CLNG-BPF [test_progs-cpuv4] bpf_gotox.bpf.o
fatal error: error in backend: Cannot select: <U+001B>[0;31mt15<U+001B>[0m: ch = brind <U+001B>[0;30mt7<U+001B>[0m, <U+001B>[0;30mt14<U+001B>[0m
  <U+001B>[0;30mt14<U+001B>[0m: i64,ch = load<(invariant load (s64) from %ir.arrayidx)> <U+001B>[0;30mt0<U+001B>[0m, <U+001B>[0;36mt13<U+001B>[0m, undef:i64, progs/bpf_gotox.c:202:8
    <U+001B>[0;36mt13<U+001B>[0m: i64 = add nuw <U+001B>[0;33mt17<U+001B>[0m, <U+001B>[0;35mt12<U+001B>[0m, progs/bpf_gotox.c:202:8
      <U+001B>[0;33mt17<U+001B>[0m: i64 = LDIMM64 TargetGlobalAddress:i64<ptr @__const.big_jump_table.jt> 0, progs/bpf_gotox.c:202:8
      <U+001B>[0;35mt12<U+001B>[0m: i64 = shl nuw nsw <U+001B>[0;32mt9<U+001B>[0m, Constant:i64<3>, progs/bpf_gotox.c:202:8
        <U+001B>[0;32mt9<U+001B>[0m: i64 = and <U+001B>[0;35mt5<U+001B>[0m, Constant:i64<255>, progs/bpf_gotox.c:202:18
          <U+001B>[0;35mt5<U+001B>[0m: i64,ch = load<(load (s64) from %ir.ctx)> <U+001B>[0;30mt0<U+001B>[0m, <U+001B>[0;32mt2<U+001B>[0m, undef:i64, progs/bpf_gotox.c:202:16
            <U+001B>[0;32mt2<U+001B>[0m: i64,ch = CopyFromReg <U+001B>[0;30mt0<U+001B>[0m, Register:i64 %5
In function: big_jump_table

CC'ing Yonghong.

[...]

Re: [PATCH v6 bpf-next 10/17] bpf, x86: add support for indirect jumps

[Anton Protopopov]

On 25/10/21 02:17PM, Eduard Zingerman wrote:
> On Sun, 2025-10-19 at 20:21 +0000, Anton Protopopov wrote:
> > Add support for a new instruction
> > 
> >     BPF_JMP|BPF_X|BPF_JA, SRC=0, DST=Rx, off=0, imm=0
> > 
> > which does an indirect jump to a location stored in Rx.  The register
> > Rx should have type PTR_TO_INSN. This new type assures that the Rx
> > register contains a value (or a range of values) loaded from a
> > correct jump table – map of type instruction array.
> > 
> > For example, for a C switch LLVM will generate the following code:
> > 
> >     0:   r3 = r1                    # "switch (r3)"
> >     1:   if r3 > 0x13 goto +0x666   # check r3 boundaries
> >     2:   r3 <<= 0x3                 # adjust to an index in array of addresses
> >     3:   r1 = 0xbeef ll             # r1 is PTR_TO_MAP_VALUE, r1->map_ptr=M
> >     5:   r1 += r3                   # r1 inherits boundaries from r3
> >     6:   r1 = *(u64 *)(r1 + 0x0)    # r1 now has type INSN_TO_PTR
> >     7:   gotox r1                   # jit will generate proper code
> > 
> > Here the gotox instruction corresponds to one particular map. This is
> > possible however to have a gotox instruction which can be loaded from
> > different maps, e.g.
> > 
> >     0:   r1 &= 0x1
> >     1:   r2 <<= 0x3
> >     2:   r3 = 0x0 ll                # load from map M_1
> >     4:   r3 += r2
> >     5:   if r1 == 0x0 goto +0x4
> >     6:   r1 <<= 0x3
> >     7:   r3 = 0x0 ll                # load from map M_2
> >     9:   r3 += r1
> >     A:   r1 = *(u64 *)(r3 + 0x0)
> >     B:   gotox r1                   # jump to target loaded from M_1 or M_2
> > 
> > During check_cfg stage the verifier will collect all the maps which
> > point to inside the subprog being verified. When building the config,
> > the high 16 bytes of the insn_state are used, so this patch
> > (theoretically) supports jump tables of up to 2^16 slots.
> > 
> > During the later stage, in check_indirect_jump, it is checked that
> > the register Rx was loaded from a particular instruction array.
> > 
> > Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
> > ---
> 
> LGTM, please, address a few remaining points.
> 
> Acked-by: Eduard Zingerman <eddyz87@gmail.com>
> 
> [...]
> 
> > diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> > index ae017c032944..d2df21fde118 100644
> > --- a/kernel/bpf/verifier.c
> > +++ b/kernel/bpf/verifier.c
> 
> [...]
> 
> > +static struct bpf_iarray *
> > +create_jt(int t, struct bpf_verifier_env *env, int fd)
>   		    	   		    	  ^^^^^^
> 				   This parameter is unused

Ah, yes, no more ->imm. Fixed.

> > +{
> > +	static struct bpf_subprog_info *subprog;
> > +	int subprog_start, subprog_end;
> > +	struct bpf_iarray *jt;
> > +	int i;
> > +
> > +	subprog = bpf_find_containing_subprog(env, t);
> > +	subprog_start = subprog->start;
> > +	subprog_end = (subprog + 1)->start;
> > +	jt = jt_from_subprog(env, subprog_start, subprog_end);
> > +	if (IS_ERR(jt))
> > +		return jt;
> > +
> > +	/* Check that the every element of the jump table fits within the given subprogram */
> > +	for (i = 0; i < jt->cnt; i++) {
> > +		if (jt->items[i] < subprog_start || jt->items[i] >= subprog_end) {
> > +			verbose(env, "jump table for insn %d points outside of the subprog [%u,%u]",
> > +					t, subprog_start, subprog_end);
> > +			return ERR_PTR(-EINVAL);
> > +		}
> > +	}
> > +
> > +	return jt;
> > +}
> 
> [...]
> 
> > +/* gotox *dst_reg */
> > +static int check_indirect_jump(struct bpf_verifier_env *env, struct bpf_insn *insn)
> > +{
> > +	struct bpf_verifier_state *other_branch;
> > +	struct bpf_reg_state *dst_reg;
> > +	struct bpf_map *map;
> > +	u32 min_index, max_index;
> > +	int err = 0;
> > +	int n;
> > +	int i;
> > +
> > +	dst_reg = reg_state(env, insn->dst_reg);
> > +	if (dst_reg->type != PTR_TO_INSN) {
> > +		verbose(env, "R%d has type %d, expected PTR_TO_INSN\n",
> > +			     insn->dst_reg, dst_reg->type);
> > +		return -EINVAL;
> > +	}
> > +
> > +	map = dst_reg->map_ptr;
> > +	if (verifier_bug_if(!map, env, "R%d has an empty map pointer", insn->dst_reg))
> > +		return -EFAULT;
> > +
> > +	if (verifier_bug_if(map->map_type != BPF_MAP_TYPE_INSN_ARRAY, env,
> > +			    "R%d has incorrect map type %d", insn->dst_reg, map->map_type))
> > +		return -EFAULT;
> 
> Nit: we discussed this in v5, let's drop the verifier_bug_if() and
>      return -EINVAL?

So, I think this is a verifier bug. We've checked above that the
register is PTR_TO_INSN, so it must have map and map type should
be BPF_MAP_TYPE_INSN_ARRAY. Now this is always true, for future
I added these warnings, just in case. Wdyt?

>      > The program can be written in a way, such that e.g. hash map
>      > pointer is passed as a parameter for gotox, that would be an
>      > incorrect program, not a verifier bug.
> 
>      Also, use reg_type_str() instead of "type %d"?

This is map type, not register? Ah, you maybe meant the first
message, I will fix it, thanks.

> > +
> > +	err = indirect_jump_min_max_index(env, insn->dst_reg, map, &min_index, &max_index);
> > +	if (err)
> > +		return err;
> > +
> > +	/* Ensure that the buffer is large enough */
> > +	if (!env->gotox_tmp_buf || env->gotox_tmp_buf->cnt < max_index - min_index + 1) {
> > +		env->gotox_tmp_buf = iarray_realloc(env->gotox_tmp_buf,
> > +						    max_index - min_index + 1);
> > +		if (!env->gotox_tmp_buf)
> > +			return -ENOMEM;
> > +	}
> > +
> > +	n = copy_insn_array_uniq(map, min_index, max_index, env->gotox_tmp_buf->items);
> 
> Nit: let's not forget about a follow-up to remove this allocation.

Thanks, in the list of followups.

> > +	if (n < 0)
> > +		return n;
> > +	if (n == 0) {
> > +		verbose(env, "register R%d doesn't point to any offset in map id=%d\n",
> > +			     insn->dst_reg, map->id);
> > +		return -EINVAL;
> > +	}
> > +
> > +	for (i = 0; i < n - 1; i++) {
> > +		other_branch = push_stack(env, env->gotox_tmp_buf->items[i],
> > +					  env->insn_idx, env->cur_state->speculative);
> > +		if (IS_ERR(other_branch))
> > +			return PTR_ERR(other_branch);
> > +	}
> > +	env->insn_idx = env->gotox_tmp_buf->items[n-1];
> > +	return 0;
> > +}
> > +
> 
> [...]

Re: [PATCH v6 bpf-next 10/17] bpf, x86: add support for indirect jumps

[Eduard Zingerman]

On Wed, 2025-10-22 at 06:51 +0000, Anton Protopopov wrote:

[...]

> > > +/* gotox *dst_reg */
> > > +static int check_indirect_jump(struct bpf_verifier_env *env, struct bpf_insn *insn)
> > > +{
> > > +	struct bpf_verifier_state *other_branch;
> > > +	struct bpf_reg_state *dst_reg;
> > > +	struct bpf_map *map;
> > > +	u32 min_index, max_index;
> > > +	int err = 0;
> > > +	int n;
> > > +	int i;
> > > +
> > > +	dst_reg = reg_state(env, insn->dst_reg);
> > > +	if (dst_reg->type != PTR_TO_INSN) {
> > > +		verbose(env, "R%d has type %d, expected PTR_TO_INSN\n",
> > > +			     insn->dst_reg, dst_reg->type);
> > > +		return -EINVAL;
> > > +	}
> > > +
> > > +	map = dst_reg->map_ptr;
> > > +	if (verifier_bug_if(!map, env, "R%d has an empty map pointer", insn->dst_reg))
> > > +		return -EFAULT;
> > > +
> > > +	if (verifier_bug_if(map->map_type != BPF_MAP_TYPE_INSN_ARRAY, env,
> > > +			    "R%d has incorrect map type %d", insn->dst_reg, map->map_type))
> > > +		return -EFAULT;
> > 
> > Nit: we discussed this in v5, let's drop the verifier_bug_if() and
> >      return -EINVAL?
> 
> So, I think this is a verifier bug. We've checked above that the
> register is PTR_TO_INSN, so it must have map and map type should
> be BPF_MAP_TYPE_INSN_ARRAY. Now this is always true, for future
> I added these warnings, just in case. Wdyt?

Wellp, yes this is a verifier bug, thank you for explaining.
Sorry for the noise.

> >      > The program can be written in a way, such that e.g. hash map
> >      > pointer is passed as a parameter for gotox, that would be an
> >      > incorrect program, not a verifier bug.
> > 
> >      Also, use reg_type_str() instead of "type %d"?
> 
> This is map type, not register? Ah, you maybe meant the first
> message, I will fix it, thanks.

Hm, right, doesn't apply here, but helps with the first message.
It looks like we don't have a similar function for maps.
But that's a "bug", not verifier message, so probably fine.

> > > +
> > > +	err = indirect_jump_min_max_index(env, insn->dst_reg, map, &min_index, &max_index);
> > > +	if (err)
> > > +		return err;
> > > +

[...]

Re: [PATCH v6 bpf-next 17/17] selftests/bpf: add C-level selftests for indirect jumps

[Anton Protopopov]

On 25/10/21 05:27PM, Eduard Zingerman wrote:
> On Sun, 2025-10-19 at 20:21 +0000, Anton Protopopov wrote:
> 
> [...]
> 
> > diff --git a/tools/testing/selftests/bpf/prog_tests/bpf_gotox.c b/tools/testing/selftests/bpf/prog_tests/bpf_gotox.c
> > new file mode 100644
> > index 000000000000..4394654ac75a
> > --- /dev/null
> > +++ b/tools/testing/selftests/bpf/prog_tests/bpf_gotox.c
> > @@ -0,0 +1,185 @@
> > +// SPDX-License-Identifier: GPL-2.0
> > +
> > +#include <test_progs.h>
> > +
> > +#include <linux/if_ether.h>
> > +#include <linux/in.h>
> > +#include <linux/ip.h>
> > +#include <linux/ipv6.h>
> > +#include <linux/in6.h>
> > +#include <linux/udp.h>
> > +#include <linux/tcp.h>
> > +
> > +#include <sys/syscall.h>
> > +#include <bpf/bpf.h>
> > +
> > +#include "bpf_gotox.skel.h"
> > +
> > +/* Disable tests for now, as CI runs with LLVM-20 */
> > +#if 0
> 
> I removed all the `#if 0` and tried to compile this test using LLVM at commit
> 1a9aba29b09e ("[RISCV] Remove unreachable break statements. NFC (#164481)").
> 
> LLVM refuses to compile showing errors like below:
> 
>   CLNG-BPF [test_progs-cpuv4] bpf_gotox.bpf.o
> fatal error: error in backend: Cannot select: <U+001B>[0;31mt15<U+001B>[0m: ch = brind <U+001B>[0;30mt7<U+001B>[0m, <U+001B>[0;30mt14<U+001B>[0m
>   <U+001B>[0;30mt14<U+001B>[0m: i64,ch = load<(invariant load (s64) from %ir.arrayidx)> <U+001B>[0;30mt0<U+001B>[0m, <U+001B>[0;36mt13<U+001B>[0m, undef:i64, progs/bpf_gotox.c:202:8
>     <U+001B>[0;36mt13<U+001B>[0m: i64 = add nuw <U+001B>[0;33mt17<U+001B>[0m, <U+001B>[0;35mt12<U+001B>[0m, progs/bpf_gotox.c:202:8
>       <U+001B>[0;33mt17<U+001B>[0m: i64 = LDIMM64 TargetGlobalAddress:i64<ptr @__const.big_jump_table.jt> 0, progs/bpf_gotox.c:202:8
>       <U+001B>[0;35mt12<U+001B>[0m: i64 = shl nuw nsw <U+001B>[0;32mt9<U+001B>[0m, Constant:i64<3>, progs/bpf_gotox.c:202:8
>         <U+001B>[0;32mt9<U+001B>[0m: i64 = and <U+001B>[0;35mt5<U+001B>[0m, Constant:i64<255>, progs/bpf_gotox.c:202:18
>           <U+001B>[0;35mt5<U+001B>[0m: i64,ch = load<(load (s64) from %ir.ctx)> <U+001B>[0;30mt0<U+001B>[0m, <U+001B>[0;32mt2<U+001B>[0m, undef:i64, progs/bpf_gotox.c:202:16
>             <U+001B>[0;32mt2<U+001B>[0m: i64,ch = CopyFromReg <U+001B>[0;30mt0<U+001B>[0m, Register:i64 %5
> In function: big_jump_table
> 
> CC'ing Yonghong.

Looks like no -mcpu=v4. I had to hack Makefile (set v3 -> v4) to build it.

Even when one tries to build test_progs-cpuv4, the v3 version is
also built.  In the bpf prog this is easy to ifdef it: x86_64 +
__clang + one can define a variable like BPF_CPU=4 in Makefile.
But the prog_tests can't easily see was the skeleton compiled
with v3 or v4, so I just ended up with putting #if 0 there for now...

> [...]

Re: [PATCH v6 bpf-next 05/17] selftests/bpf: add selftests for new insn_array map

[Anton Protopopov]

On 25/10/21 04:51PM, Eduard Zingerman wrote:
> On Sun, 2025-10-19 at 20:21 +0000, Anton Protopopov wrote:
> > Add the following selftests for new insn_array map:
> > 
> >   * Incorrect instruction indexes are rejected
> >   * Two programs can't use the same map
> >   * BPF progs can't operate the map
> >   * no changes to code => map is the same
> >   * expected changes when instructions are added
> >   * expected changes when instructions are deleted
> >   * expected changes when multiple functions are present
> > 
> > Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
> > ---
> 
> Acked-by: Eduard Zingerman <eddyz87@gmail.com>
> 
> >  .../selftests/bpf/prog_tests/bpf_insn_array.c | 404 ++++++++++++++++++
> >  1 file changed, 404 insertions(+)
> >  create mode 100644 tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c
> > 
> > diff --git a/tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c b/tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c
> > new file mode 100644
> > index 000000000000..a4304ef5be13
> > --- /dev/null
> > +++ b/tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c
> 
> [...]
> 
> > +static void check_bpf_no_lookup(void)
> 
> This one can be moved to prog_tests/bpf_insn_array.c, I think.

A typo? (This is a patch for the prog_tests/bpf_insn_array.c)

> > +{
> > +	struct bpf_insn insns[] = {
> > +		BPF_LD_MAP_FD(BPF_REG_1, 0),
> > +		BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0),
> > +		BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
> > +		BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8),
> > +		BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem),
> > +		BPF_EXIT_INSN(),
> > +	};
> > +	int prog_fd = -1, map_fd;
> > +
> > +	map_fd = map_create(BPF_MAP_TYPE_INSN_ARRAY, 1);
> > +	if (!ASSERT_GE(map_fd, 0, "map_create"))
> > +		return;
> > +
> > +	insns[0].imm = map_fd;
> > +
> > +	if (!ASSERT_EQ(bpf_map_freeze(map_fd), 0, "bpf_map_freeze"))
> > +		goto cleanup;
> > +
> > +	prog_fd = prog_load(insns, ARRAY_SIZE(insns), NULL, 0);
> > +	if (!ASSERT_EQ(prog_fd, -EINVAL, "program should have been rejected (prog_fd != -EINVAL)"))
> > +		goto cleanup;
> > +
> > +	/* correctness: check that prog is still loadable with normal map */
> > +	close(map_fd);
> > +	map_fd = map_create(BPF_MAP_TYPE_ARRAY, 1);
> > +	insns[0].imm = map_fd;
> > +	prog_fd = prog_load(insns, ARRAY_SIZE(insns), NULL, 0);
> > +	if (!ASSERT_GE(prog_fd, 0, "bpf(BPF_PROG_LOAD)"))
> > +		goto cleanup;
> > +
> > +cleanup:
> > +	close(prog_fd);
> > +	close(map_fd);
> > +}
> 
> [...]

Re: [PATCH v6 bpf-next 05/17] selftests/bpf: add selftests for new insn_array map

[Eduard Zingerman]

On Wed, 2025-10-22 at 13:44 +0000, Anton Protopopov wrote:
> On 25/10/21 04:51PM, Eduard Zingerman wrote:
> > On Sun, 2025-10-19 at 20:21 +0000, Anton Protopopov wrote:
> > > Add the following selftests for new insn_array map:
> > > 
> > >   * Incorrect instruction indexes are rejected
> > >   * Two programs can't use the same map
> > >   * BPF progs can't operate the map
> > >   * no changes to code => map is the same
> > >   * expected changes when instructions are added
> > >   * expected changes when instructions are deleted
> > >   * expected changes when multiple functions are present
> > > 
> > > Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
> > > ---
> > 
> > Acked-by: Eduard Zingerman <eddyz87@gmail.com>
> > 
> > >  .../selftests/bpf/prog_tests/bpf_insn_array.c | 404 ++++++++++++++++++
> > >  1 file changed, 404 insertions(+)
> > >  create mode 100644 tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c
> > > 
> > > diff --git a/tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c b/tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c
> > > new file mode 100644
> > > index 000000000000..a4304ef5be13
> > > --- /dev/null
> > > +++ b/tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c
> > 
> > [...]
> > 
> > > +static void check_bpf_no_lookup(void)
> > 
> > This one can be moved to prog_tests/bpf_insn_array.c, I think.
> 
> A typo? (This is a patch for the prog_tests/bpf_insn_array.c)

Yes, I mean progs/verifier_gotox.c, the one with inline assembly.

> 
> > > +{
> > > +	struct bpf_insn insns[] = {
> > > +		BPF_LD_MAP_FD(BPF_REG_1, 0),
> > > +		BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0),
> > > +		BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
> > > +		BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8),
> > > +		BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem),
> > > +		BPF_EXIT_INSN(),
> > > +	};
> > > +	int prog_fd = -1, map_fd;
> > > +
> > > +	map_fd = map_create(BPF_MAP_TYPE_INSN_ARRAY, 1);
> > > +	if (!ASSERT_GE(map_fd, 0, "map_create"))
> > > +		return;
> > > +
> > > +	insns[0].imm = map_fd;
> > > +
> > > +	if (!ASSERT_EQ(bpf_map_freeze(map_fd), 0, "bpf_map_freeze"))
> > > +		goto cleanup;
> > > +
> > > +	prog_fd = prog_load(insns, ARRAY_SIZE(insns), NULL, 0);
> > > +	if (!ASSERT_EQ(prog_fd, -EINVAL, "program should have been rejected (prog_fd != -EINVAL)"))
> > > +		goto cleanup;
> > > +
> > > +	/* correctness: check that prog is still loadable with normal map */
> > > +	close(map_fd);
> > > +	map_fd = map_create(BPF_MAP_TYPE_ARRAY, 1);
> > > +	insns[0].imm = map_fd;
> > > +	prog_fd = prog_load(insns, ARRAY_SIZE(insns), NULL, 0);
> > > +	if (!ASSERT_GE(prog_fd, 0, "bpf(BPF_PROG_LOAD)"))
> > > +		goto cleanup;
> > > +
> > > +cleanup:
> > > +	close(prog_fd);
> > > +	close(map_fd);
> > > +}
> > 
> > [...]

Re: [PATCH v6 bpf-next 05/17] selftests/bpf: add selftests for new insn_array map

[Eduard Zingerman]

On Wed, 2025-10-22 at 14:06 +0000, Anton Protopopov wrote:

[...]

> > > > > diff --git a/tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c b/tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c
> > > > > new file mode 100644
> > > > > index 000000000000..a4304ef5be13
> > > > > --- /dev/null
> > > > > +++ b/tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c
> > > > 
> > > > [...]
> > > > 
> > > > > +static void check_bpf_no_lookup(void)
> > > > 
> > > > This one can be moved to prog_tests/bpf_insn_array.c, I think.
> > > 
> > > A typo? (This is a patch for the prog_tests/bpf_insn_array.c)
> > 
> > Yes, I mean progs/verifier_gotox.c, the one with inline assembly.
> 
> I think it should stay here. There will be other usages of the
> instruction array, and neither should allow operations on it from
> a BPF prog (indirect calls, static keys).

It will be functionally identical and like 3x-4x time shorter in that file.

Re: [PATCH v6 bpf-next 05/17] selftests/bpf: add selftests for new insn_array map

[Anton Protopopov]

On 25/10/22 06:55AM, Eduard Zingerman wrote:
> On Wed, 2025-10-22 at 13:44 +0000, Anton Protopopov wrote:
> > On 25/10/21 04:51PM, Eduard Zingerman wrote:
> > > On Sun, 2025-10-19 at 20:21 +0000, Anton Protopopov wrote:
> > > > Add the following selftests for new insn_array map:
> > > > 
> > > >   * Incorrect instruction indexes are rejected
> > > >   * Two programs can't use the same map
> > > >   * BPF progs can't operate the map
> > > >   * no changes to code => map is the same
> > > >   * expected changes when instructions are added
> > > >   * expected changes when instructions are deleted
> > > >   * expected changes when multiple functions are present
> > > > 
> > > > Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
> > > > ---
> > > 
> > > Acked-by: Eduard Zingerman <eddyz87@gmail.com>
> > > 
> > > >  .../selftests/bpf/prog_tests/bpf_insn_array.c | 404 ++++++++++++++++++
> > > >  1 file changed, 404 insertions(+)
> > > >  create mode 100644 tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c
> > > > 
> > > > diff --git a/tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c b/tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c
> > > > new file mode 100644
> > > > index 000000000000..a4304ef5be13
> > > > --- /dev/null
> > > > +++ b/tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c
> > > 
> > > [...]
> > > 
> > > > +static void check_bpf_no_lookup(void)
> > > 
> > > This one can be moved to prog_tests/bpf_insn_array.c, I think.
> > 
> > A typo? (This is a patch for the prog_tests/bpf_insn_array.c)
> 
> Yes, I mean progs/verifier_gotox.c, the one with inline assembly.

I think it should stay here. There will be other usages of the
instruction array, and neither should allow operations on it from
a BPF prog (indirect calls, static keys).

> > 
> > > > +{
> > > > +	struct bpf_insn insns[] = {
> > > > +		BPF_LD_MAP_FD(BPF_REG_1, 0),
> > > > +		BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0),
> > > > +		BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
> > > > +		BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8),
> > > > +		BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem),
> > > > +		BPF_EXIT_INSN(),
> > > > +	};
> > > > +	int prog_fd = -1, map_fd;
> > > > +
> > > > +	map_fd = map_create(BPF_MAP_TYPE_INSN_ARRAY, 1);
> > > > +	if (!ASSERT_GE(map_fd, 0, "map_create"))
> > > > +		return;
> > > > +
> > > > +	insns[0].imm = map_fd;
> > > > +
> > > > +	if (!ASSERT_EQ(bpf_map_freeze(map_fd), 0, "bpf_map_freeze"))
> > > > +		goto cleanup;
> > > > +
> > > > +	prog_fd = prog_load(insns, ARRAY_SIZE(insns), NULL, 0);
> > > > +	if (!ASSERT_EQ(prog_fd, -EINVAL, "program should have been rejected (prog_fd != -EINVAL)"))
> > > > +		goto cleanup;
> > > > +
> > > > +	/* correctness: check that prog is still loadable with normal map */
> > > > +	close(map_fd);
> > > > +	map_fd = map_create(BPF_MAP_TYPE_ARRAY, 1);
> > > > +	insns[0].imm = map_fd;
> > > > +	prog_fd = prog_load(insns, ARRAY_SIZE(insns), NULL, 0);
> > > > +	if (!ASSERT_GE(prog_fd, 0, "bpf(BPF_PROG_LOAD)"))
> > > > +		goto cleanup;
> > > > +
> > > > +cleanup:
> > > > +	close(prog_fd);
> > > > +	close(map_fd);
> > > > +}
> > > 
> > > [...]

Re: [PATCH v6 bpf-next 05/17] selftests/bpf: add selftests for new insn_array map

[Anton Protopopov]

On 25/10/22 07:03AM, Eduard Zingerman wrote:
> On Wed, 2025-10-22 at 14:06 +0000, Anton Protopopov wrote:
> 
> [...]
> 
> > > > > > diff --git a/tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c b/tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c
> > > > > > new file mode 100644
> > > > > > index 000000000000..a4304ef5be13
> > > > > > --- /dev/null
> > > > > > +++ b/tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c
> > > > > 
> > > > > [...]
> > > > > 
> > > > > > +static void check_bpf_no_lookup(void)
> > > > > 
> > > > > This one can be moved to prog_tests/bpf_insn_array.c, I think.
> > > > 
> > > > A typo? (This is a patch for the prog_tests/bpf_insn_array.c)
> > > 
> > > Yes, I mean progs/verifier_gotox.c, the one with inline assembly.
> > 
> > I think it should stay here. There will be other usages of the
> > instruction array, and neither should allow operations on it from
> > a BPF prog (indirect calls, static keys).
> 
> It will be functionally identical and like 3x-4x time shorter in that file.

Ok, I see. I will either move it there, or add another
verifier_insn_array (if I see how to convert/create more "insn array
level" tests like this).

Re: [PATCH v6 bpf-next 05/17] selftests/bpf: add selftests for new insn_array map

[Eduard Zingerman]

On Wed, 2025-10-22 at 07:03 -0700, Eduard Zingerman wrote:
> On Wed, 2025-10-22 at 14:06 +0000, Anton Protopopov wrote:
> 
> [...]
> 
> > > > > > diff --git a/tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c b/tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c
> > > > > > new file mode 100644
> > > > > > index 000000000000..a4304ef5be13
> > > > > > --- /dev/null
> > > > > > +++ b/tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c
> > > > > 
> > > > > [...]
> > > > > 
> > > > > > +static void check_bpf_no_lookup(void)
> > > > > 
> > > > > This one can be moved to prog_tests/bpf_insn_array.c, I think.
> > > > 
> > > > A typo? (This is a patch for the prog_tests/bpf_insn_array.c)
> > > 
> > > Yes, I mean progs/verifier_gotox.c, the one with inline assembly.
> > 
> > I think it should stay here. There will be other usages of the
> > instruction array, and neither should allow operations on it from
> > a BPF prog (indirect calls, static keys).
> 
> It will be functionally identical and like 3x-4x time shorter in that file.

Wait. I'm being stupid again. There is no simple way to get an FD for
the iarray map when writing a bpf C. So your current implementation
makes most sense. Please disregard this nitpick, the patch is good as
it is.

Re: [PATCH v6 bpf-next 16/17] selftests/bpf: add new verifier_gotox test

[Anton Protopopov]

On 25/10/21 03:42PM, Eduard Zingerman wrote:
> On Sun, 2025-10-19 at 20:21 +0000, Anton Protopopov wrote:
> > Add a set of tests to validate core gotox functionality
> > without need to rely on compilers.
> > 
> > Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
> > ---
> 
> Thank you for adding these.
> Could you please also add a test cases that checks the following errors:
> - "jump table for insn %d points outside of the subprog [%u,%u]"
> - "the sum of R%u umin_value %llu and off %u is too big\n"
> - "register R%d doesn't point to any offset in map id=%d\n"

Yeah, sorry, these actually were on my list, but I've postponed them
for the next version. Will add. (I also need to add a few selftests
on the offset when loading from map.)

> 
> Might be the case that some of these can't be triggered because of the
> check_mem_access() call.
> 
> [...]
> 
> > diff --git a/tools/testing/selftests/bpf/progs/verifier_gotox.c b/tools/testing/selftests/bpf/progs/verifier_gotox.c
> > new file mode 100644
> > index 000000000000..1a92e4d321e8
> > --- /dev/null
> > +++ b/tools/testing/selftests/bpf/progs/verifier_gotox.c
> 
> [...]
> 
> > +/*
> > + * Gotox is forbidden when there is no jump table loaded
> > + * which points to the sub-function where the gotox is used
> > + */
> > +SEC("socket")
> > +__failure __msg("no jump tables found for subprog starting at 0")
>                                                               ^^^^
> 				Nit: one day we need to figure out a way to
> 				     report subprogram names, when reporting
> 				     check_cfg() errors.

But those are not always present, right?

> 
> > +__naked void jump_table_no_jump_table(void)
> > +{
> > +	asm volatile ("						\
> > +	.8byte %[gotox_r0];					\
> > +	r0 = 1;							\
> > +	exit;							\
> > +"	:							\
> > +	: __imm_insn(gotox_r0, BPF_RAW_INSN(BPF_JMP | BPF_JA | BPF_X, BPF_REG_0, 0, 0 , 0))
> > +	: __clobber_all);
> > +}
> > +
> > +/*
> > + * Incorrect type of the target register, only PTR_TO_INSN allowed
> > + */
> > +SEC("socket")
> > +__failure __msg("R1 has type 1, expected PTR_TO_INSN")
>                            ^^^^^^
> 	      log.c:reg_type_str() should help here.

Yes, thanks, this was changed to address your comment
in the other patch.

> > +__naked void jump_table_incorrect_dst_reg_type(void)
> > +{
> > +	asm volatile ("						\
> > +	.pushsection .jumptables,\"\",@progbits;		\
> > +jt0_%=:								\
> > +	.quad ret0_%=;						\
> > +	.quad ret1_%=;						\
> > +	.size jt0_%=, 16;					\
> > +	.global jt0_%=;						\
> > +	.popsection;						\
> > +								\
> > +	r0 = jt0_%= ll;						\
> > +	r0 += 8;						\
> > +	r0 = *(u64 *)(r0 + 0);					\
> > +	r1 = 42;						\
> > +	.8byte %[gotox_r1];					\
> > +	ret0_%=:						\
> > +	r0 = 0;							\
> > +	exit;							\
> > +	ret1_%=:						\
> > +	r0 = 1;							\
> > +	exit;							\
> > +"	:							\
> > +	: __imm_insn(gotox_r1, BPF_RAW_INSN(BPF_JMP | BPF_JA | BPF_X, BPF_REG_1, 0, 0 , 0))
> > +	: __clobber_all);
> > +}
> > +
> > +#define DEFINE_INVALID_SIZE_PROG(READ_SIZE, OUTCOME)			\
> 
> Nit: this can be merged with DEFINE_SIMPLE_JUMP_TABLE_PROG.

Didn't want to overload the macro too much so the prog stays
readable. (Here are two different regs are used.) I will check
how it looks like if I merge them, and merge, if it looks ok-ish.

> > +									\
> > +	SEC("socket")							\
> > +	OUTCOME								\
> > +	__naked void jump_table_invalid_read_size_ ## READ_SIZE(void)	\
> > +	{								\
> > +		asm volatile ("						\
> > +		.pushsection .jumptables,\"\",@progbits;		\
> > +	jt0_%=:								\
> > +		.quad ret0_%=;						\
> > +		.quad ret1_%=;						\
> > +		.size jt0_%=, 16;					\
> > +		.global jt0_%=;						\
> > +		.popsection;						\
> > +									\
> > +		r0 = jt0_%= ll;						\
> > +		r0 += 8;						\
> > +		r0 = *(" #READ_SIZE " *)(r0 + 0);			\
> > +		.8byte %[gotox_r0];					\
> > +		ret0_%=:						\
> > +		r0 = 0;							\
> > +		exit;							\
> > +		ret1_%=:						\
> > +		r0 = 1;							\
> > +		exit;							\
> > +	"	:							\
> > +		: __imm_insn(gotox_r0, BPF_RAW_INSN(BPF_JMP | BPF_JA | BPF_X, BPF_REG_0, 0, 0 , 0)) \
> > +		: __clobber_all);					\
> > +	}
> > +
> > +DEFINE_INVALID_SIZE_PROG(u32, __failure __msg("Invalid read of 4 bytes from insn_array"))
> > +DEFINE_INVALID_SIZE_PROG(u16, __failure __msg("Invalid read of 2 bytes from insn_array"))
> > +DEFINE_INVALID_SIZE_PROG(u8,  __failure __msg("Invalid read of 1 bytes from insn_array"))
> 
> [...]

Re: [PATCH v6 bpf-next 04/17] bpf, x86: add new map type: instructions array

[Anton Protopopov]

On 25/10/21 04:26PM, Eduard Zingerman wrote:
> On Sun, 2025-10-19 at 20:21 +0000, Anton Protopopov wrote:
> 
> [...]
> 
> > The functionality provided by this patch will be extended in consequent
> > patches to implement BPF Static Keys, indirect jumps, and indirect calls.
> > 
> > Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
> > ---
> 
> Aside from what Alexei pointed out, I only have a nit regarding jitted_ip.
> 
> Reviewed-by: Eduard Zingerman <eddyz87@gmail.com>
> 
> [...]
> 
> > diff --git a/include/linux/bpf.h b/include/linux/bpf.h
> > index e53cda0aabb6..363355628d2e 100644
> > --- a/include/linux/bpf.h
> > +++ b/include/linux/bpf.h
> > @@ -3789,4 +3789,40 @@ int bpf_prog_get_file_line(struct bpf_prog *prog, unsigned long ip, const char *
> >  			   const char **linep, int *nump);
> 
> [...]
> 
> > +/*
> > + * The struct bpf_insn_ptr structure describes a pointer to a
> > + * particular instruction in a loaded BPF program. Initially
> > + * it is initialised from userspace via user_value.xlated_off.
> > + * During the program verification all other fields are populated
> > + * accordingly:
> > + *
> > + *   jitted_ip:       address of the instruction in the jitted image
> > + *   user_value:      user-visible original, xlated, and jitted offsets
> > + */
> > +struct bpf_insn_ptr {
> > +	void *jitted_ip;
> 
> I think this one is no longer used anywhere.
> I see it set in bpf_prog_update_insn_ptr() but it is not read anywhere.

Nice, thanks :)

> > +	struct bpf_insn_array_value user_value;
> > +};
> > +
> 
> [...]
> 
> > diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
> > index 6829936d33f5..805d441363cd 100644
> 
> [...]
> 
> > @@ -7645,4 +7646,24 @@ enum bpf_kfunc_flags {
> >  	BPF_F_PAD_ZEROS = (1ULL << 0),
> >  };
> >  
> > +/*
> > + * Values of a BPF_MAP_TYPE_INSN_ARRAY entry must be of this type.
> > + *
> > + * Before the map is used the orig_off field should point to an
> > + * instruction inside the program being loaded. The other fields
> > + * must be set to 0.
> > + *
> > + * After the program is loaded, the xlated_off will be adjusted
> > + * by the verifier to point to the index of the original instruction
> > + * in the xlated program. If the instruction is deleted, it will
> > + * be set to (u32)-1. The jitted_off will be set to the corresponding
> > + * offset in the jitted image of the program.
> > + */
> > +struct bpf_insn_array_value {
> > +	__u32 orig_off;
> > +	__u32 xlated_off;
> > +	__u32 jitted_off;
> > +	__u32 :32;
> 
> This :u32, is it for alignment or for future extensibility?

Both. We maybe might have some flags added in future for different use cases...

> > +
> 
> [...]
> 
> > diff --git a/kernel/bpf/bpf_insn_array.c b/kernel/bpf/bpf_insn_array.c
> > new file mode 100644
> 
> [...]
> 
> > @@ -0,0 +1,288 @@
> > +// SPDX-License-Identifier: GPL-2.0-only
> > +/* Copyright (c) 2025 Isovalent */
> > +
> > +#include <linux/bpf.h>
> > +
> > +#define MAX_INSN_ARRAY_ENTRIES 256
> > +
> > +struct bpf_insn_array {
> > +	struct bpf_map map;
> > +	atomic_t used;
> > +	long *ips;
> > +	DECLARE_FLEX_ARRAY(struct bpf_insn_ptr, ptrs);
> > +};
> 
> [...]
> 
> > +static inline u32 insn_array_alloc_size(u32 max_entries)
> > +{
> > +	const u32 base_size = sizeof(struct bpf_insn_array);
> > +	const u32 entry_size = sizeof(struct bpf_insn_ptr);
> > +
> > +	return base_size + entry_size * max_entries;
> > +}
> 
> Since this is doing a flexible array thing anyway, maybe also include
> size for `ips` here? And in insn_array_alloc() point it at the tail
> area.

Yes, great idea.

> [...]

Re: [PATCH v6 bpf-next 14/17] libbpf: support llvm-generated indirect jumps

[Anton Protopopov]

On 25/10/21 03:18PM, Eduard Zingerman wrote:
> On Sun, 2025-10-19 at 20:21 +0000, Anton Protopopov wrote:
> 
> [...]
> 
> > ---
> >  tools/lib/bpf/libbpf.c        | 240 +++++++++++++++++++++++++++++++++-
> >  tools/lib/bpf/libbpf_probes.c |   4 +
> >  tools/lib/bpf/linker.c        |  10 +-
> >  3 files changed, 251 insertions(+), 3 deletions(-)
> > 
> > diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
> > index b90574f39d1c..ee44bc49a3ba 100644
> > --- a/tools/lib/bpf/libbpf.c
> > +++ b/tools/lib/bpf/libbpf.c
> 
> [...]
> 
> > +/*
> > + * In LLVM the .jumptables section contains jump tables entries relative to the
> > + * section start. The BPF kernel-side code expects jump table offsets relative
> > + * to the beginning of the program (passed in bpf(BPF_PROG_LOAD)). This helper
> > + * computes a delta to be added when creating a map.
> > + */
> > +static int jt_adjust_off(struct bpf_program *prog, int insn_idx)
> > +{
> > +	int i;
> > +
> > +	for (i = prog->subprog_cnt - 1; i >= 0; i--) {
> > +		if (insn_idx >= prog->subprogs[i].sub_insn_off)
> 
> Sorry, I'm still confused about what happens here.
> The `insn_idx` is comes from relocation, meaning that it is a value
> recorded relative to section start, right?  On the other hand,
> `.sub_insn_off` is an offset of a subprogram within a concatenated
> program, about to be loaded.  These values should not be compared
> directly.
> 
> I think, that my suggestion from v5 [1] should be easier to understand:

Well, if you insist :) (I saw the next e-mail as well, thanks.)

>    > Or rename this thing to find_subprog_idx(), pass relo object into
>    > create_jt_map(), call find_subprog_idx() there, and do the following:
>    >
>    >   xlated_off = jt[i] / sizeof(struct bpf_insn);
>    >   /* make xlated_off relative to subprogram start */
>    >   xlated_off -= prog->subprogs[subprog_idx].sec_insn_off;
>    >   /* make xlated_off relative to main subprogram start */
>    >   xlated_off += prog->subprogs[subprog_idx].sub_insn_off;
> 
> [1] https://lore.kernel.org/bpf/b5fd31c3e703c8c84c6710f5536510fbce04b36f.camel@gmail.com/
> 
> > +			return prog->subprogs[i].sub_insn_off - prog->subprogs[i].sec_insn_off;
> > +	}
> > +
> > +	return -prog->sec_insn_off;
> > +}
> > +
> > +
> >  /* Relocate data references within program code:
> >   *  - map references;
> >   *  - global variable references;
> > @@ -6235,6 +6422,21 @@ bpf_object__relocate_data(struct bpf_object *obj, struct bpf_program *prog)
> >  		case RELO_CORE:
> >  			/* will be handled by bpf_program_record_relos() */
> >  			break;
> > +		case RELO_INSN_ARRAY: {
> > +			int map_fd;
> > +
> > +			map_fd = create_jt_map(obj, prog, relo->sym_off, relo->sym_size,
> > +					       jt_adjust_off(prog, relo->insn_idx));
> > +			if (map_fd < 0) {
> > +				pr_warn("prog '%s': relo #%d: can't create jump table: sym_off %u\n",
> > +						prog->name, i, relo->sym_off);
> > +				return map_fd;
> > +			}
> > +			insn[0].src_reg = BPF_PSEUDO_MAP_VALUE;
> > +			insn->imm = map_fd;
> > +			insn->off = 0;
> > +		}
> > +			break;
> >  		default:
> >  			pr_warn("prog '%s': relo #%d: bad relo type %d\n",
> >  				prog->name, i, relo->type);
> 
> [...]
> 
> > @@ -9228,6 +9457,15 @@ void bpf_object__close(struct bpf_object *obj)
> >  
> >  	zfree(&obj->arena_data);
> >  
> > +	zfree(&obj->jumptables_data);
> > +	obj->jumptables_data_sz = 0;
> > +
> > +	if (obj->jumptable_maps && obj->jumptable_map_cnt) {
> 
> Nit: outer 'if' seems unnecessary.

I suspect this was a check for if obj->jumptable_maps is null or not.
I think this should never happen that jumptable_map_cnt && !jumptable_map,
so I will remove the if.

> > +		for (i = 0; i < obj->jumptable_map_cnt; i++)
> > +			close(obj->jumptable_maps[i].fd);
> > +	}
> > +	zfree(&obj->jumptable_maps);
> > +
> >  	free(obj);
> >  }
> 
> [...]
> 
> > diff --git a/tools/lib/bpf/linker.c b/tools/lib/bpf/linker.c
> > index 56ae77047bc3..3defd4bc9154 100644
> > --- a/tools/lib/bpf/linker.c
> > +++ b/tools/lib/bpf/linker.c
> > @@ -27,6 +27,8 @@
> >  #include "strset.h"
> >  
> >  #define BTF_EXTERN_SEC ".extern"
> > +#define JUMPTABLES_SEC ".jumptables"
> > +#define JUMPTABLES_REL_SEC ".rel.jumptables"
> 
> Nit: maybe avoid duplicating JUMPTABLES_SEC by moving all *_SEC macro
>      to libbpf_internal.h?

Yes, ok.

> >  
> >  struct src_sec {
> >  	const char *sec_name;
> > @@ -2025,6 +2027,9 @@ static int linker_append_elf_sym(struct bpf_linker *linker, struct src_obj *obj,
> >  			obj->sym_map[src_sym_idx] = dst_sec->sec_sym_idx;
> >  			return 0;
> >  		}
> > +
> > +		if (strcmp(src_sec->sec_name, JUMPTABLES_SEC) == 0)
> > +			goto add_sym;
> >  	}
> >  
> >  	if (sym_bind == STB_LOCAL)
> > @@ -2271,8 +2276,9 @@ static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob
> >  						insn->imm += sec->dst_off / sizeof(struct bpf_insn);
> >  					else
> >  						insn->imm += sec->dst_off;
> > -				} else {
> > -					pr_warn("relocation against STT_SECTION in non-exec section is not supported!\n");
> > +				} else if (strcmp(src_sec->sec_name, JUMPTABLES_REL_SEC) != 0) {
> > +					pr_warn("relocation against STT_SECTION in section %s is not supported!\n",
> > +						src_sec->sec_name);
> 
> Sorry, I missed this on a previous iteration.
> LLVM generates section relative offsets for jump table contents, so it
> seems that relocations inside jump table section should not occur.
> Is this a leftover, or am I confused?

I think this is a leftover in LLVM, so I have to keep it here.
I will check again with the latest LLVM.

> >  					return -EINVAL;
> >  				}
> >  			}

Re: [PATCH v6 bpf-next 14/17] libbpf: support llvm-generated indirect jumps

[Anton Protopopov]

On 25/10/24 12:52PM, Anton Protopopov wrote:
> On 25/10/21 03:18PM, Eduard Zingerman wrote:
> > On Sun, 2025-10-19 at 20:21 +0000, Anton Protopopov wrote:
> > 
> > [...]
> > 
> > > ---
> > >  tools/lib/bpf/libbpf.c        | 240 +++++++++++++++++++++++++++++++++-
> > >  tools/lib/bpf/libbpf_probes.c |   4 +
> > >  tools/lib/bpf/linker.c        |  10 +-
> > >  3 files changed, 251 insertions(+), 3 deletions(-)
> > > 
> > > diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
> > > index b90574f39d1c..ee44bc49a3ba 100644
> > > --- a/tools/lib/bpf/libbpf.c
> > > +++ b/tools/lib/bpf/libbpf.c
> > 
> > [...]
> > 
> > > +/*
> > > + * In LLVM the .jumptables section contains jump tables entries relative to the
> > > + * section start. The BPF kernel-side code expects jump table offsets relative
> > > + * to the beginning of the program (passed in bpf(BPF_PROG_LOAD)). This helper
> > > + * computes a delta to be added when creating a map.
> > > + */
> > > +static int jt_adjust_off(struct bpf_program *prog, int insn_idx)
> > > +{
> > > +	int i;
> > > +
> > > +	for (i = prog->subprog_cnt - 1; i >= 0; i--) {
> > > +		if (insn_idx >= prog->subprogs[i].sub_insn_off)
> > 
> > Sorry, I'm still confused about what happens here.
> > The `insn_idx` is comes from relocation, meaning that it is a value
> > recorded relative to section start, right?  On the other hand,
> > `.sub_insn_off` is an offset of a subprogram within a concatenated
> > program, about to be loaded.  These values should not be compared
> > directly.
> > 
> > I think, that my suggestion from v5 [1] should be easier to understand:
> 
> Well, if you insist :) (I saw the next e-mail as well, thanks.)
> 
> >    > Or rename this thing to find_subprog_idx(), pass relo object into
> >    > create_jt_map(), call find_subprog_idx() there, and do the following:
> >    >
> >    >   xlated_off = jt[i] / sizeof(struct bpf_insn);
> >    >   /* make xlated_off relative to subprogram start */
> >    >   xlated_off -= prog->subprogs[subprog_idx].sec_insn_off;
> >    >   /* make xlated_off relative to main subprogram start */
> >    >   xlated_off += prog->subprogs[subprog_idx].sub_insn_off;
> > 
> > [1] https://lore.kernel.org/bpf/b5fd31c3e703c8c84c6710f5536510fbce04b36f.camel@gmail.com/
> > 
> > > +			return prog->subprogs[i].sub_insn_off - prog->subprogs[i].sec_insn_off;
> > > +	}
> > > +
> > > +	return -prog->sec_insn_off;
> > > +}
> > > +
> > > +
> > >  /* Relocate data references within program code:
> > >   *  - map references;
> > >   *  - global variable references;
> > > @@ -6235,6 +6422,21 @@ bpf_object__relocate_data(struct bpf_object *obj, struct bpf_program *prog)
> > >  		case RELO_CORE:
> > >  			/* will be handled by bpf_program_record_relos() */
> > >  			break;
> > > +		case RELO_INSN_ARRAY: {
> > > +			int map_fd;
> > > +
> > > +			map_fd = create_jt_map(obj, prog, relo->sym_off, relo->sym_size,
> > > +					       jt_adjust_off(prog, relo->insn_idx));
> > > +			if (map_fd < 0) {
> > > +				pr_warn("prog '%s': relo #%d: can't create jump table: sym_off %u\n",
> > > +						prog->name, i, relo->sym_off);
> > > +				return map_fd;
> > > +			}
> > > +			insn[0].src_reg = BPF_PSEUDO_MAP_VALUE;
> > > +			insn->imm = map_fd;
> > > +			insn->off = 0;
> > > +		}
> > > +			break;
> > >  		default:
> > >  			pr_warn("prog '%s': relo #%d: bad relo type %d\n",
> > >  				prog->name, i, relo->type);
> > 
> > [...]
> > 
> > > @@ -9228,6 +9457,15 @@ void bpf_object__close(struct bpf_object *obj)
> > >  
> > >  	zfree(&obj->arena_data);
> > >  
> > > +	zfree(&obj->jumptables_data);
> > > +	obj->jumptables_data_sz = 0;
> > > +
> > > +	if (obj->jumptable_maps && obj->jumptable_map_cnt) {
> > 
> > Nit: outer 'if' seems unnecessary.
> 
> I suspect this was a check for if obj->jumptable_maps is null or not.
> I think this should never happen that jumptable_map_cnt && !jumptable_map,
> so I will remove the if.
> 
> > > +		for (i = 0; i < obj->jumptable_map_cnt; i++)
> > > +			close(obj->jumptable_maps[i].fd);
> > > +	}
> > > +	zfree(&obj->jumptable_maps);
> > > +
> > >  	free(obj);
> > >  }
> > 
> > [...]
> > 
> > > diff --git a/tools/lib/bpf/linker.c b/tools/lib/bpf/linker.c
> > > index 56ae77047bc3..3defd4bc9154 100644
> > > --- a/tools/lib/bpf/linker.c
> > > +++ b/tools/lib/bpf/linker.c
> > > @@ -27,6 +27,8 @@
> > >  #include "strset.h"
> > >  
> > >  #define BTF_EXTERN_SEC ".extern"
> > > +#define JUMPTABLES_SEC ".jumptables"
> > > +#define JUMPTABLES_REL_SEC ".rel.jumptables"
> > 
> > Nit: maybe avoid duplicating JUMPTABLES_SEC by moving all *_SEC macro
> >      to libbpf_internal.h?
> 
> Yes, ok.
> 
> > >  
> > >  struct src_sec {
> > >  	const char *sec_name;
> > > @@ -2025,6 +2027,9 @@ static int linker_append_elf_sym(struct bpf_linker *linker, struct src_obj *obj,
> > >  			obj->sym_map[src_sym_idx] = dst_sec->sec_sym_idx;
> > >  			return 0;
> > >  		}
> > > +
> > > +		if (strcmp(src_sec->sec_name, JUMPTABLES_SEC) == 0)
> > > +			goto add_sym;
> > >  	}
> > >  
> > >  	if (sym_bind == STB_LOCAL)
> > > @@ -2271,8 +2276,9 @@ static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob
> > >  						insn->imm += sec->dst_off / sizeof(struct bpf_insn);
> > >  					else
> > >  						insn->imm += sec->dst_off;
> > > -				} else {
> > > -					pr_warn("relocation against STT_SECTION in non-exec section is not supported!\n");
> > > +				} else if (strcmp(src_sec->sec_name, JUMPTABLES_REL_SEC) != 0) {
> > > +					pr_warn("relocation against STT_SECTION in section %s is not supported!\n",
> > > +						src_sec->sec_name);
> > 
> > Sorry, I missed this on a previous iteration.
> > LLVM generates section relative offsets for jump table contents, so it
> > seems that relocations inside jump table section should not occur.
> > Is this a leftover, or am I confused?
> 
> I think this is a leftover in LLVM, so I have to keep it here.
> I will check again with the latest LLVM.

Yes, the latest LLVM I've built (llvmorg-22-init-12400-g09eea2256e53)
does generate it:

    $ llvm-readelf -S bpf_gotox.bpf.o | grep jumptables
      [ 8] .jumptables       PROGBITS        ...
      [ 9] .rel.jumptables   REL             ...

> > >  					return -EINVAL;
> > >  				}
> > >  			}

Re: [PATCH v6 bpf-next 17/17] selftests/bpf: add C-level selftests for indirect jumps

[Anton Protopopov]

On 25/10/22 01:34PM, Anton Protopopov wrote:
> On 25/10/21 05:27PM, Eduard Zingerman wrote:
> > On Sun, 2025-10-19 at 20:21 +0000, Anton Protopopov wrote:
> > 
> > [...]
> > 
> > > diff --git a/tools/testing/selftests/bpf/prog_tests/bpf_gotox.c b/tools/testing/selftests/bpf/prog_tests/bpf_gotox.c
> > > new file mode 100644
> > > index 000000000000..4394654ac75a
> > > --- /dev/null
> > > +++ b/tools/testing/selftests/bpf/prog_tests/bpf_gotox.c
> > > @@ -0,0 +1,185 @@
> > > +// SPDX-License-Identifier: GPL-2.0
> > > +
> > > +#include <test_progs.h>
> > > +
> > > +#include <linux/if_ether.h>
> > > +#include <linux/in.h>
> > > +#include <linux/ip.h>
> > > +#include <linux/ipv6.h>
> > > +#include <linux/in6.h>
> > > +#include <linux/udp.h>
> > > +#include <linux/tcp.h>
> > > +
> > > +#include <sys/syscall.h>
> > > +#include <bpf/bpf.h>
> > > +
> > > +#include "bpf_gotox.skel.h"
> > > +
> > > +/* Disable tests for now, as CI runs with LLVM-20 */
> > > +#if 0
> > 
> > I removed all the `#if 0` and tried to compile this test using LLVM at commit
> > 1a9aba29b09e ("[RISCV] Remove unreachable break statements. NFC (#164481)").
> > 
> > LLVM refuses to compile showing errors like below:
> > 
> >   CLNG-BPF [test_progs-cpuv4] bpf_gotox.bpf.o
> > fatal error: error in backend: Cannot select: <U+001B>[0;31mt15<U+001B>[0m: ch = brind <U+001B>[0;30mt7<U+001B>[0m, <U+001B>[0;30mt14<U+001B>[0m
> >   <U+001B>[0;30mt14<U+001B>[0m: i64,ch = load<(invariant load (s64) from %ir.arrayidx)> <U+001B>[0;30mt0<U+001B>[0m, <U+001B>[0;36mt13<U+001B>[0m, undef:i64, progs/bpf_gotox.c:202:8
> >     <U+001B>[0;36mt13<U+001B>[0m: i64 = add nuw <U+001B>[0;33mt17<U+001B>[0m, <U+001B>[0;35mt12<U+001B>[0m, progs/bpf_gotox.c:202:8
> >       <U+001B>[0;33mt17<U+001B>[0m: i64 = LDIMM64 TargetGlobalAddress:i64<ptr @__const.big_jump_table.jt> 0, progs/bpf_gotox.c:202:8
> >       <U+001B>[0;35mt12<U+001B>[0m: i64 = shl nuw nsw <U+001B>[0;32mt9<U+001B>[0m, Constant:i64<3>, progs/bpf_gotox.c:202:8
> >         <U+001B>[0;32mt9<U+001B>[0m: i64 = and <U+001B>[0;35mt5<U+001B>[0m, Constant:i64<255>, progs/bpf_gotox.c:202:18
> >           <U+001B>[0;35mt5<U+001B>[0m: i64,ch = load<(load (s64) from %ir.ctx)> <U+001B>[0;30mt0<U+001B>[0m, <U+001B>[0;32mt2<U+001B>[0m, undef:i64, progs/bpf_gotox.c:202:16
> >             <U+001B>[0;32mt2<U+001B>[0m: i64,ch = CopyFromReg <U+001B>[0;30mt0<U+001B>[0m, Register:i64 %5
> > In function: big_jump_table
> > 
> > CC'ing Yonghong.
> 
> Looks like no -mcpu=v4. I had to hack Makefile (set v3 -> v4) to build it.
> 
> Even when one tries to build test_progs-cpuv4, the v3 version is
> also built.  In the bpf prog this is easy to ifdef it: x86_64 +
> __clang + one can define a variable like BPF_CPU=4 in Makefile.
> But the prog_tests can't easily see was the skeleton compiled
> with v3 or v4, so I just ended up with putting #if 0 there for now...

Yes, checked with the latest LLVM that this is "v3 vs. v4".
If you pass -mcpu=v4 the code will compile fine.

> > [...]

Re: [PATCH v6 bpf-next 16/17] selftests/bpf: add new verifier_gotox test

[Anton Protopopov]

On 25/10/24 11:40AM, Anton Protopopov wrote:
> On 25/10/21 03:42PM, Eduard Zingerman wrote:
> > On Sun, 2025-10-19 at 20:21 +0000, Anton Protopopov wrote:
> > > Add a set of tests to validate core gotox functionality
> > > without need to rely on compilers.
> > > 
> > > Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
> > > ---
> > 
> > Thank you for adding these.
> > Could you please also add a test cases that checks the following errors:
> > - "jump table for insn %d points outside of the subprog [%u,%u]"
> > - "the sum of R%u umin_value %llu and off %u is too big\n"
> > - "register R%d doesn't point to any offset in map id=%d\n"
> 
> Yeah, sorry, these actually were on my list, but I've postponed them
> for the next version. Will add. (I also need to add a few selftests
> on the offset when loading from map.)

So, tbh, I can't actually find a way to trigger any of them,
looks like these conditions are always caught earlier...

> > 
> > Might be the case that some of these can't be triggered because of the
> > check_mem_access() call.
> > 
> > [...]
> > 
> > > diff --git a/tools/testing/selftests/bpf/progs/verifier_gotox.c b/tools/testing/selftests/bpf/progs/verifier_gotox.c
> > > new file mode 100644
> > > index 000000000000..1a92e4d321e8
> > > --- /dev/null
> > > +++ b/tools/testing/selftests/bpf/progs/verifier_gotox.c
> > 
> > [...]
> > 
> > > +/*
> > > + * Gotox is forbidden when there is no jump table loaded
> > > + * which points to the sub-function where the gotox is used
> > > + */
> > > +SEC("socket")
> > > +__failure __msg("no jump tables found for subprog starting at 0")
> >                                                               ^^^^
> > 				Nit: one day we need to figure out a way to
> > 				     report subprogram names, when reporting
> > 				     check_cfg() errors.
> 
> But those are not always present, right?
> 
> > 
> > > +__naked void jump_table_no_jump_table(void)
> > > +{
> > > +	asm volatile ("						\
> > > +	.8byte %[gotox_r0];					\
> > > +	r0 = 1;							\
> > > +	exit;							\
> > > +"	:							\
> > > +	: __imm_insn(gotox_r0, BPF_RAW_INSN(BPF_JMP | BPF_JA | BPF_X, BPF_REG_0, 0, 0 , 0))
> > > +	: __clobber_all);
> > > +}
> > > +
> > > +/*
> > > + * Incorrect type of the target register, only PTR_TO_INSN allowed
> > > + */
> > > +SEC("socket")
> > > +__failure __msg("R1 has type 1, expected PTR_TO_INSN")
> >                            ^^^^^^
> > 	      log.c:reg_type_str() should help here.
> 
> Yes, thanks, this was changed to address your comment
> in the other patch.
> 
> > > +__naked void jump_table_incorrect_dst_reg_type(void)
> > > +{
> > > +	asm volatile ("						\
> > > +	.pushsection .jumptables,\"\",@progbits;		\
> > > +jt0_%=:								\
> > > +	.quad ret0_%=;						\
> > > +	.quad ret1_%=;						\
> > > +	.size jt0_%=, 16;					\
> > > +	.global jt0_%=;						\
> > > +	.popsection;						\
> > > +								\
> > > +	r0 = jt0_%= ll;						\
> > > +	r0 += 8;						\
> > > +	r0 = *(u64 *)(r0 + 0);					\
> > > +	r1 = 42;						\
> > > +	.8byte %[gotox_r1];					\
> > > +	ret0_%=:						\
> > > +	r0 = 0;							\
> > > +	exit;							\
> > > +	ret1_%=:						\
> > > +	r0 = 1;							\
> > > +	exit;							\
> > > +"	:							\
> > > +	: __imm_insn(gotox_r1, BPF_RAW_INSN(BPF_JMP | BPF_JA | BPF_X, BPF_REG_1, 0, 0 , 0))
> > > +	: __clobber_all);
> > > +}
> > > +
> > > +#define DEFINE_INVALID_SIZE_PROG(READ_SIZE, OUTCOME)			\
> > 
> > Nit: this can be merged with DEFINE_SIMPLE_JUMP_TABLE_PROG.
> 
> Didn't want to overload the macro too much so the prog stays
> readable. (Here are two different regs are used.) I will check
> how it looks like if I merge them, and merge, if it looks ok-ish.
> 
> > > +									\
> > > +	SEC("socket")							\
> > > +	OUTCOME								\
> > > +	__naked void jump_table_invalid_read_size_ ## READ_SIZE(void)	\
> > > +	{								\
> > > +		asm volatile ("						\
> > > +		.pushsection .jumptables,\"\",@progbits;		\
> > > +	jt0_%=:								\
> > > +		.quad ret0_%=;						\
> > > +		.quad ret1_%=;						\
> > > +		.size jt0_%=, 16;					\
> > > +		.global jt0_%=;						\
> > > +		.popsection;						\
> > > +									\
> > > +		r0 = jt0_%= ll;						\
> > > +		r0 += 8;						\
> > > +		r0 = *(" #READ_SIZE " *)(r0 + 0);			\
> > > +		.8byte %[gotox_r0];					\
> > > +		ret0_%=:						\
> > > +		r0 = 0;							\
> > > +		exit;							\
> > > +		ret1_%=:						\
> > > +		r0 = 1;							\
> > > +		exit;							\
> > > +	"	:							\
> > > +		: __imm_insn(gotox_r0, BPF_RAW_INSN(BPF_JMP | BPF_JA | BPF_X, BPF_REG_0, 0, 0 , 0)) \
> > > +		: __clobber_all);					\
> > > +	}
> > > +
> > > +DEFINE_INVALID_SIZE_PROG(u32, __failure __msg("Invalid read of 4 bytes from insn_array"))
> > > +DEFINE_INVALID_SIZE_PROG(u16, __failure __msg("Invalid read of 2 bytes from insn_array"))
> > > +DEFINE_INVALID_SIZE_PROG(u8,  __failure __msg("Invalid read of 1 bytes from insn_array"))
> > 
> > [...]

Re: [PATCH v6 bpf-next 16/17] selftests/bpf: add new verifier_gotox test

[Eduard Zingerman]

On Sun, 2025-10-26 at 12:34 +0000, Anton Protopopov wrote:
> On 25/10/24 11:40AM, Anton Protopopov wrote:
> > On 25/10/21 03:42PM, Eduard Zingerman wrote:
> > > On Sun, 2025-10-19 at 20:21 +0000, Anton Protopopov wrote:
> > > > Add a set of tests to validate core gotox functionality
> > > > without need to rely on compilers.
> > > > 
> > > > Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
> > > > ---
> > > 
> > > Thank you for adding these.
> > > Could you please also add a test cases that checks the following errors:
> > > - "jump table for insn %d points outside of the subprog [%u,%u]"

I'm surprised this one can't be triggered.
In case if there are multiple subprograms defined,
and map has two entries pointing to different subprograms,
what other checks can catch this?
I see only the check in the bpf_insn_array_init() that will make sure
that offset does not point outside of the program.

> > > - "the sum of R%u umin_value %llu and off %u is too big\n"

Ok, this one is handled by map value read.

> > > - "register R%d doesn't point to any offset in map id=%d\n"

And this one as well.

> > Yeah, sorry, these actually were on my list, but I've postponed them
> > for the next version. Will add. (I also need to add a few selftests
> > on the offset when loading from map.)
> 
> So, tbh, I can't actually find a way to trigger any of them,
> looks like these conditions are always caught earlier...

Re: [PATCH v6 bpf-next 16/17] selftests/bpf: add new verifier_gotox test

[Anton Protopopov]

On 25/10/27 04:11PM, Eduard Zingerman wrote:
> On Sun, 2025-10-26 at 12:34 +0000, Anton Protopopov wrote:
> > On 25/10/24 11:40AM, Anton Protopopov wrote:
> > > On 25/10/21 03:42PM, Eduard Zingerman wrote:
> > > > On Sun, 2025-10-19 at 20:21 +0000, Anton Protopopov wrote:
> > > > > Add a set of tests to validate core gotox functionality
> > > > > without need to rely on compilers.
> > > > > 
> > > > > Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
> > > > > ---
> > > > 
> > > > Thank you for adding these.
> > > > Could you please also add a test cases that checks the following errors:
> > > > - "jump table for insn %d points outside of the subprog [%u,%u]"
> 
> I'm surprised this one can't be triggered.
> In case if there are multiple subprograms defined,
> and map has two entries pointing to different subprograms,
> what other checks can catch this?
> I see only the check in the bpf_insn_array_init() that will make sure
> that offset does not point outside of the program.

Yes, you were correct. I will add a test.

> > > > - "the sum of R%u umin_value %llu and off %u is too big\n"
> 
> Ok, this one is handled by map value read.
> 
> > > > - "register R%d doesn't point to any offset in map id=%d\n"
> 
> And this one as well.
> 
> > > Yeah, sorry, these actually were on my list, but I've postponed them
> > > for the next version. Will add. (I also need to add a few selftests
> > > on the offset when loading from map.)
> > 
> > So, tbh, I can't actually find a way to trigger any of them,
> > looks like these conditions are always caught earlier...